[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:48 dist-test-slave-m01t krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:48 dist-test-slave-m01t krb5kdc[399](info): set up 2 sockets
Sep 24 05:03:48 dist-test-slave-m01t krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 05:03:49.645624 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:49 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:49.749181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:49.824023 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:49.824235 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:49.824265 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:49.825397 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:49.825435 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:49.825452 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:49.825469 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:49.825484 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:49.826896 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:49.827132 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:49.827350 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:03:49.832350 412 server_base.cc:1034] running on GCE node
W20260924 05:03:49.832319 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:49.832324 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:49.832347 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:49.832756 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:49.833045 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:49.834187 412 hybrid_clock.cc:648] HybridClock initialized: now 1790226229834179 us; error 26 us; skew 500 ppm
Sep 24 05:03:49 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:49.836500 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:49.836947 412 webserver.cc:460] Webserver started at http://127.0.98.62:41109/ using document root <none> and password file <none>
I20260924 05:03:49.837142 412 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:49.837190 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:49.837324 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:49.838428 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "174b0ef918884ecf8dd33d7eb8c2b247"
format_stamp: "Formatted at 2026-09-24 05:03:49 on dist-test-slave-m01t"
server_key: "2051792be8ec21d6404b31ebdd64af3e"
server_key_iv: "8d32a691f2a1496d55446279a9b8683a"
server_key_version: "encryptionkey@0"
I20260924 05:03:49.838778 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "174b0ef918884ecf8dd33d7eb8c2b247"
format_stamp: "Formatted at 2026-09-24 05:03:49 on dist-test-slave-m01t"
server_key: "2051792be8ec21d6404b31ebdd64af3e"
server_key_iv: "8d32a691f2a1496d55446279a9b8683a"
server_key_version: "encryptionkey@0"
I20260924 05:03:49.840045 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:49.840684 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:49.840857 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:49.840939 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "174b0ef918884ecf8dd33d7eb8c2b247"
format_stamp: "Formatted at 2026-09-24 05:03:49 on dist-test-slave-m01t"
server_key: "2051792be8ec21d6404b31ebdd64af3e"
server_key_iv: "8d32a691f2a1496d55446279a9b8683a"
server_key_version: "encryptionkey@0"
I20260924 05:03:49.840994 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:49.862685 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:49.864327 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:49.864437 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:49.867836 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39301
I20260924 05:03:49.867878 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39301 every 8 connection(s)
I20260924 05:03:49.868191 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:49.868978 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:49.871074 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247: Bootstrap starting.
I20260924 05:03:49.871582 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:49.871845 480 log.cc:824] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:49.872419 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247: No bootstrap required, opened a new log
I20260924 05:03:49.872771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 412
I20260924 05:03:49.872864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:49.873438 392 external_mini_cluster.cc:1428] Setting key 0a7b5301c2c60bfc6a611bc1f74e8514
I20260924 05:03:49.874014 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260924 05:03:49.874155 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:49.874240 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174b0ef918884ecf8dd33d7eb8c2b247, State: Initialized, Role: FOLLOWER
I20260924 05:03:49.874388 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260924 05:03:49.874469 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:49.874491 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:49.874524 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:49.875355 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260924 05:03:49.875454 480 leader_election.cc:302] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 174b0ef918884ecf8dd33d7eb8c2b247; no voters:
I20260924 05:03:49.875597 480 leader_election.cc:288] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:49.875643 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:49.875847 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [term 1 LEADER]: Becoming Leader. State: Replica: 174b0ef918884ecf8dd33d7eb8c2b247, State: Running, Role: LEADER
I20260924 05:03:49.875893 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:49.876003 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260924 05:03:49.876312 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 174b0ef918884ecf8dd33d7eb8c2b247. Latest consensus state: current_term: 1 leader_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } } }
I20260924 05:03:49.876322 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174b0ef918884ecf8dd33d7eb8c2b247" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } } }
I20260924 05:03:49.876386 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:49.876415 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:49.876555 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:49.877032 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:49.878629 492 catalog_manager.cc:1347] Generated new cluster ID: ef6b530bbc214a5ca3613b53acce5457
I20260924 05:03:49.878688 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:03:49 dist-test-slave-m01t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:49.886796 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:49.887224 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:49.874205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.0.98.62:39301)
0924 05:03:49.874385 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:49.874394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:49.874594 (+ 200us) server_negotiation.cc:407] Connection header received
0924 05:03:49.876672 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:49.876679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:49.876731 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:49.876834 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:49.878772 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:49.879647 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:49.880604 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:49.881004 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:49.881095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:49.884107 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:49.884111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:49.884111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:49.884121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:49.885506 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:49.885925 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:49.885927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:49.885928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:49.886015 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:49.886248 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:49.886249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:49.886249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:49.886438 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:49.886571 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:49.886884 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 05:03:49.887005 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260924 05:03:49.887465 492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:49.896008 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 174b0ef918884ecf8dd33d7eb8c2b247: Generated new TSK 0
I20260924 05:03:49.896176 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:03:49.922443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:49.999032 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:49.999228 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:49.999249 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:50.000344 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:50.000382 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:50.000407 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:03:50.001835 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:50.002035 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:50.002252 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:50.006669 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:50.006664 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:50.006790 506 server_base.cc:1034] running on GCE node
W20260924 05:03:50.006664 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:50.006992 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:50.007205 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:50.008330 506 hybrid_clock.cc:648] HybridClock initialized: now 1790226230008318 us; error 26 us; skew 500 ppm
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:50.010466 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:03:50.010833 506 webserver.cc:460] Webserver started at http://127.0.98.1:36547/ using document root <none> and password file <none>
I20260924 05:03:50.011008 506 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:50.011051 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:50.011169 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:50.012116 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "6ef1bc99c5b247afa3c7045d50ef6324"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "3bdda961896bd01a4bd19fb325d4d32c"
server_key_iv: "bd7a4cd870f245872f24fc2d15aa8d81"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.012436 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6ef1bc99c5b247afa3c7045d50ef6324"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "3bdda961896bd01a4bd19fb325d4d32c"
server_key_iv: "bd7a4cd870f245872f24fc2d15aa8d81"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.013559 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:03:50.014190 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.014330 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.014402 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "6ef1bc99c5b247afa3c7045d50ef6324"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "3bdda961896bd01a4bd19fb325d4d32c"
server_key_iv: "bd7a4cd870f245872f24fc2d15aa8d81"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.014462 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:50.027508 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:50.029254 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:50.029389 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:50.029614 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:03:50.029966 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:50.030000 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.030030 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:50.030047 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.035344 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38397
I20260924 05:03:50.035401 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38397 every 8 connection(s)
I20260924 05:03:50.035689 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:03:50.036350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 506
I20260924 05:03:50.036438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:03:50.036578 392 external_mini_cluster.cc:1428] Setting key 11f7834ba341fa3061fbb5990ffef906
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:03:50.044955 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.036346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50633 (local address 127.0.98.62:39301)
0924 05:03:50.036404 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:50.036406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:50.036583 (+ 177us) server_negotiation.cc:407] Connection header received
0924 05:03:50.037104 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:50.037104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:50.037118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:50.037168 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:50.038161 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.038708 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.039751 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.039929 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.039971 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.042318 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:50.042319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:50.042320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:50.042326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:50.043598 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.044014 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.044015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.044016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.044062 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.044337 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.044337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.044337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.044433 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:50.044479 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:50.044798 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 05:03:50.044886 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:03:50.045158 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.036305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39301 (local address 127.0.98.1:50633)
0924 05:03:50.036465 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:50.036507 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:50.037021 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:50.037246 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:50.037251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:50.037860 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:50.038054 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:50.038060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.038788 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.038789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:50.039690 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:50.039692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.039995 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.039997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:50.040136 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.040626 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:50.040636 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:50.042232 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:50.043666 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:50.043672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:50.043675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:50.043947 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:50.044117 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:50.044118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:50.044118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:50.044279 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:50.044489 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:50.044496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:50.044673 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 05:03:50.044784 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 05:03:50.044945 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260924 05:03:50.045567 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39301
I20260924 05:03:50.045660 635 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:50.045862 635 heartbeater.cc:511] Master 127.0.98.62:39301 requested a full tablet report, sending...
I20260924 05:03:50.046346 444 ts_manager.cc:194] Registered new tserver with Master: 6ef1bc99c5b247afa3c7045d50ef6324 (127.0.98.1:38397)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 05:03:50.046988 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:03:50.058122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:50.133987 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:50.134202 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:50.134233 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:50.135381 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:50.135426 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:50.135460 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:03:50.136883 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:50.137130 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:50.137339 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:50.141944 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:50.141932 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:50.142428 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:50.142503 640 server_base.cc:1034] running on GCE node
I20260924 05:03:50.142692 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:50.142902 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:50.144037 640 hybrid_clock.cc:648] HybridClock initialized: now 1790226230144024 us; error 27 us; skew 500 ppm
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:50.146245 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:03:50.146652 640 webserver.cc:460] Webserver started at http://127.0.98.2:35381/ using document root <none> and password file <none>
I20260924 05:03:50.146847 640 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:50.146893 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:50.147007 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:50.147998 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "43ea0da3a59a4ef6a47e2f22a7d08bee"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "c8fdbc0c80408ff62a2c72de267f70b5"
server_key_iv: "90c6087bde48423b40fc04c7fb4895ab"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.148343 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43ea0da3a59a4ef6a47e2f22a7d08bee"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "c8fdbc0c80408ff62a2c72de267f70b5"
server_key_iv: "90c6087bde48423b40fc04c7fb4895ab"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.149551 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:50.150213 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.150357 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.150431 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "43ea0da3a59a4ef6a47e2f22a7d08bee"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "c8fdbc0c80408ff62a2c72de267f70b5"
server_key_iv: "90c6087bde48423b40fc04c7fb4895ab"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.150493 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:50.162359 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:50.163901 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:50.164022 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:50.164212 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:03:50.164474 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:50.164503 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.164528 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:50.164546 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.170027 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36639
I20260924 05:03:50.170085 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36639 every 8 connection(s)
I20260924 05:03:50.170348 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:03:50.171355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 640
I20260924 05:03:50.171434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:03:50.171581 392 external_mini_cluster.cc:1428] Setting key e2d79626aa6aa5dc000658f40c555a9f
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:03:50.178910 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.170943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60389 (local address 127.0.98.62:39301)
0924 05:03:50.170983 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:50.170985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:50.171164 (+ 179us) server_negotiation.cc:407] Connection header received
0924 05:03:50.171673 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:50.171674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:50.171688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:50.171737 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:50.172594 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.173082 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.174051 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.174259 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.174314 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.176428 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:50.176429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:50.176429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:50.176437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:50.177735 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.178067 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.178068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.178068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.178121 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.178328 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.178329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.178330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.178436 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:50.178489 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:50.178746 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 05:03:50.178824 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:03:50.179059 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.170924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39301 (local address 127.0.98.2:60389)
0924 05:03:50.171064 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:50.171102 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:50.171574 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:50.171801 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:50.171805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:50.172316 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:50.172504 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:50.172508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.173161 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.173162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:50.173980 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:50.173982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.174325 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.174326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:50.174435 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.174904 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:50.174911 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:50.176347 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:50.177799 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:50.177805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:50.177807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:50.178006 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:50.178170 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:50.178171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:50.178171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:50.178276 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:50.178487 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:50.178493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:50.178633 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 05:03:50.178718 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 05:03:50.178858 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260924 05:03:50.179445 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39301
I20260924 05:03:50.179538 769 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:50.179705 769 heartbeater.cc:511] Master 127.0.98.62:39301 requested a full tablet report, sending...
I20260924 05:03:50.180063 444 ts_manager.cc:194] Registered new tserver with Master: 43ea0da3a59a4ef6a47e2f22a7d08bee (127.0.98.2:36639)
I20260924 05:03:50.180436 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:03:50.192335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:50.270782 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:50.270973 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:50.271003 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:50.272100 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:50.272142 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:50.272176 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:03:50.273803 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:50.274035 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:50.274269 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:50.279222 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:50.279385 774 server_base.cc:1034] running on GCE node
W20260924 05:03:50.279723 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:50.279767 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:50.279996 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:50.280221 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:50.281353 774 hybrid_clock.cc:648] HybridClock initialized: now 1790226230281341 us; error 28 us; skew 500 ppm
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:50.283974 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:03:50.284363 774 webserver.cc:460] Webserver started at http://127.0.98.3:40155/ using document root <none> and password file <none>
I20260924 05:03:50.284528 774 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:50.284564 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:50.284658 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:50.285565 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "3cfa56ed6981404388a996a71ce96a89"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "84a34be5c631b5a2b07e5ed99901b701"
server_key_iv: "06262305a7957614a848cdf060edf396"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.285898 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3cfa56ed6981404388a996a71ce96a89"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "84a34be5c631b5a2b07e5ed99901b701"
server_key_iv: "06262305a7957614a848cdf060edf396"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.287032 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:03:50.287583 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.287703 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:03:50.287770 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "3cfa56ed6981404388a996a71ce96a89"
format_stamp: "Formatted at 2026-09-24 05:03:50 on dist-test-slave-m01t"
server_key: "84a34be5c631b5a2b07e5ed99901b701"
server_key_iv: "06262305a7957614a848cdf060edf396"
server_key_version: "encryptionkey@0"
I20260924 05:03:50.287830 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:50.300611 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:50.302609 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:50.302731 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:50.302920 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:03:50.303188 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:50.303220 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.303241 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:50.303264 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:50.308967 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33085
I20260924 05:03:50.309023 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33085 every 8 connection(s)
I20260924 05:03:50.309283 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:50.315980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 774
I20260924 05:03:50.316071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:03:50.316219 392 external_mini_cluster.cc:1428] Setting key ae8961cfec1b9f889a5474f3b32b9d2b
I20260924 05:03:50.317222 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.309896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55643 (local address 127.0.98.62:39301)
0924 05:03:50.309958 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:50.309960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:50.310111 (+ 151us) server_negotiation.cc:407] Connection header received
0924 05:03:50.310560 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:50.310561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:50.310573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:50.310623 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:50.311454 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.311937 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.312846 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.313047 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.313100 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.314864 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:50.314865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:50.314865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:50.314871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:50.315875 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.316233 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.316235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.316235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.316295 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.316508 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.316508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.316509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.316629 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:50.316692 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:50.317080 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 05:03:50.317154 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:03:50.317255 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.309868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39301 (local address 127.0.98.3:55643)
0924 05:03:50.310012 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:50.310044 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:50.310474 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:50.310695 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:50.310698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:50.311204 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:50.311372 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:50.311376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.312010 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.312011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:50.312772 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:50.312774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.313108 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.313110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:50.313214 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.313552 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:50.313559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:50.314794 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:50.315950 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:50.315955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:50.315956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:50.316174 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:50.316352 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:50.316352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:50.316353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:50.316449 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:50.316689 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:50.316695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:50.316837 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 05:03:50.316940 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 05:03:50.317056 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260924 05:03:50.317593 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39301
I20260924 05:03:50.317663 903 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:50.317830 903 heartbeater.cc:511] Master 127.0.98.62:39301 requested a full tablet report, sending...
I20260924 05:03:50.318157 444 ts_manager.cc:194] Registered new tserver with Master: 3cfa56ed6981404388a996a71ce96a89 (127.0.98.3:33085)
I20260924 05:03:50.318470 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55643
I20260924 05:03:50.328022 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:03:50.349303 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.342506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.0.98.1:38397)
0924 05:03:50.342662 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:50.342666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:50.342691 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:03:50.342716 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:50.342717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:50.342731 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:50.342785 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:50.343303 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.344034 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.344722 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.345026 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.345074 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.347021 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:50.347028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:50.347028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:50.347034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:50.348433 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.348685 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.348687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.348688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.348737 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.348850 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.348850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.348850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.348958 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:50.349070 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:50.349128 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 05:03:50.349198 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
W20260924 05:03:50.349640 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46990
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:03:50.369191 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:50.363251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.0.98.1:38397)
0924 05:03:50.363301 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:50.363303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:50.363380 (+ 77us) server_negotiation.cc:407] Connection header received
0924 05:03:50.363444 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:50.363444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:50.363455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:50.363502 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:50.363981 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.364437 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.365131 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:50.365325 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:50.365376 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:50.367299 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:50.367300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:50.367300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:50.367307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:50.368285 (+ 978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.368553 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.368554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.368555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.368608 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:50.368761 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:50.368762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:50.368762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:50.368847 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:50.368946 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:50.369031 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 05:03:50.369103 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:03:50.369614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 506
I20260924 05:03:50.373314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 640
I20260924 05:03:50.378367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 774
I20260924 05:03:50.381954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 412
2026-09-24T05:03:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1688 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[917](info): set up 2 sockets
Sep 24 05:03:50 dist-test-slave-m01t krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 05:03:51.320901 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:51.422012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:51.498216 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:51.498391 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:51.498411 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:51.499462 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:51.499493 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:51.499505 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:51.499516 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:51.499531 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:51.500954 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:51.501135 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:51.501294 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:51.505519 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.505509 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:51.505564 930 server_base.cc:1034] running on GCE node
W20260924 05:03:51.505508 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:51.505862 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:51.506119 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:51.507251 930 hybrid_clock.cc:648] HybridClock initialized: now 1790226231507238 us; error 29 us; skew 500 ppm
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:51.509414 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:51.509802 930 webserver.cc:460] Webserver started at http://127.0.98.62:43867/ using document root <none> and password file <none>
I20260924 05:03:51.509992 930 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:51.510037 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:51.510149 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:51.511053 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "527633416f3a4ce8b52f5378a8d5a747"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "3b85c3c2b4d546d6362b6263e7b4faca"
server_key_iv: "7754a95af22bd56ee8fa0e07fd4f2455"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.511351 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "527633416f3a4ce8b52f5378a8d5a747"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "3b85c3c2b4d546d6362b6263e7b4faca"
server_key_iv: "7754a95af22bd56ee8fa0e07fd4f2455"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.512446 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:03:51.513007 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.513149 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:03:51.513228 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "527633416f3a4ce8b52f5378a8d5a747"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "3b85c3c2b4d546d6362b6263e7b4faca"
server_key_iv: "7754a95af22bd56ee8fa0e07fd4f2455"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.513285 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:51.539203 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:51.541465 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:51.541610 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:51.545293 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37359
I20260924 05:03:51.545334 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37359 every 8 connection(s)
I20260924 05:03:51.545671 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:51.546507 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:51.548662 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747: Bootstrap starting.
I20260924 05:03:51.549185 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:51.549439 998 log.cc:824] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:51.550068 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747: No bootstrap required, opened a new log
I20260924 05:03:51.551638 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527633416f3a4ce8b52f5378a8d5a747" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260924 05:03:51.551770 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:51.551790 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527633416f3a4ce8b52f5378a8d5a747, State: Initialized, Role: FOLLOWER
I20260924 05:03:51.551921 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527633416f3a4ce8b52f5378a8d5a747" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260924 05:03:51.551993 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:51.552022 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:51.552069 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:51.552531 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527633416f3a4ce8b52f5378a8d5a747" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260924 05:03:51.552631 998 leader_election.cc:302] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 527633416f3a4ce8b52f5378a8d5a747; no voters:
I20260924 05:03:51.552774 998 leader_election.cc:288] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:51.552815 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:51.552994 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:51.553165 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [term 1 LEADER]: Becoming Leader. State: Replica: 527633416f3a4ce8b52f5378a8d5a747, State: Running, Role: LEADER
I20260924 05:03:51.553258 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527633416f3a4ce8b52f5378a8d5a747" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } }
I20260924 05:03:51.553571 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527633416f3a4ce8b52f5378a8d5a747. Latest consensus state: current_term: 1 leader_uuid: "527633416f3a4ce8b52f5378a8d5a747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527633416f3a4ce8b52f5378a8d5a747" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } } }
I20260924 05:03:51.553668 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:51.553570 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527633416f3a4ce8b52f5378a8d5a747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527633416f3a4ce8b52f5378a8d5a747" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37359 } } }
I20260924 05:03:51.553782 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:51.553890 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:51.554509 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:51.555805 1010 catalog_manager.cc:1347] Generated new cluster ID: 9b0462fe02cf44008ffd72d704929d1d
I20260924 05:03:51.555852 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:51.555845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 930
I20260924 05:03:51.555933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:51.556092 392 external_mini_cluster.cc:1428] Setting key 11afe9e89eff6cfc1c014849cd9ed0e0
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:51.566918 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.556493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.98.62:37359)
0924 05:03:51.556633 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:51.556642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:51.556675 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:03:51.556803 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:51.556810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:51.556861 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:51.556983 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:51.557635 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.559031 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.560060 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.560967 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.561056 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.563628 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:51.563632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:51.563632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:51.563643 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:51.565406 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.565784 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.565787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.565788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.565889 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.566060 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.566061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.566061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.566350 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:51.566483 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:51.566569 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:03:51.566698 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260924 05:03:51.574120 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:51.574703 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:51.580732 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527633416f3a4ce8b52f5378a8d5a747: Generated new TSK 0
I20260924 05:03:51.580871 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:03:51.604538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37359
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:51.680917 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:51.681093 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:51.681114 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:51.682253 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:51.682288 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:51.682314 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:03:51.683656 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:51.683858 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:51.684053 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:51.688380 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.688416 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.688382 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:51.688638 1024 server_base.cc:1034] running on GCE node
I20260924 05:03:51.688823 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:51.689040 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:51.690181 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790226231690171 us; error 27 us; skew 500 ppm
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:51.692366 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:03:51.692759 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34987/ using document root <none> and password file <none>
I20260924 05:03:51.692956 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:51.693002 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:51.693132 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:51.694147 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad35123159084357abc6fb54605307b2"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "3871008db8f3fee6702f91c1dea925af"
server_key_iv: "e27bc9632cbebb0073e7ff494fe6d9e2"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.694451 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad35123159084357abc6fb54605307b2"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "3871008db8f3fee6702f91c1dea925af"
server_key_iv: "e27bc9632cbebb0073e7ff494fe6d9e2"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.695578 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:51.696161 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.696309 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:51.696383 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "ad35123159084357abc6fb54605307b2"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "3871008db8f3fee6702f91c1dea925af"
server_key_iv: "e27bc9632cbebb0073e7ff494fe6d9e2"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.696439 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:51.707211 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:51.708695 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:51.708812 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:51.708997 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:03:51.709276 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:51.709309 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.709335 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:51.709352 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.716094 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43025
I20260924 05:03:51.716148 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43025 every 8 connection(s)
I20260924 05:03:51.716519 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:03:51.718689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1024
I20260924 05:03:51.718770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:03:51.718911 392 external_mini_cluster.cc:1428] Setting key 125b2aa792d9d4cc5a05bbebf4830f85
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:03:51.726069 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.717145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47527 (local address 127.0.98.62:37359)
0924 05:03:51.717198 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:51.717200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:51.717589 (+ 389us) server_negotiation.cc:407] Connection header received
0924 05:03:51.718244 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:51.718245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:51.718259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:51.718330 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:51.719381 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.719907 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.721129 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.721344 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.721400 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.723419 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:51.723420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:51.723421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:51.723428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:51.724662 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.725143 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.725145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.725145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.725202 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.725409 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.725410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.725410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.725529 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:51.725585 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:51.725882 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 05:03:51.725979 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:03:51.726224 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.717143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37359 (local address 127.0.98.1:47527)
0924 05:03:51.717460 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:51.717503 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:51.718102 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:51.718399 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:51.718403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:51.719061 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:51.719298 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:51.719303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.719991 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.719992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:51.721050 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:51.721052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.721409 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.721410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:51.721540 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.722021 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:51.722031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:51.723335 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:51.724834 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:51.724841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:51.724844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:51.725073 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:51.725255 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:51.725256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:51.725256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:51.725356 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:51.725584 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:51.725591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:51.725754 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 05:03:51.725856 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 05:03:51.726010 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":71,"threads_started":1}
I20260924 05:03:51.726682 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37359
I20260924 05:03:51.726769 1153 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:51.726912 1153 heartbeater.cc:511] Master 127.0.98.62:37359 requested a full tablet report, sending...
I20260924 05:03:51.727408 962 ts_manager.cc:194] Registered new tserver with Master: ad35123159084357abc6fb54605307b2 (127.0.98.1:43025)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 05:03:51.728096 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:03:51.740712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37359
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:51.817096 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:51.817271 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:51.817302 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:51.818446 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:51.818485 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:51.818519 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:03:51.819876 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:51.820098 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:51.820294 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:51.824810 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.824806 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.824846 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:51.824893 1158 server_base.cc:1034] running on GCE node
I20260924 05:03:51.825194 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:51.825421 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:51.826547 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790226231826536 us; error 27 us; skew 500 ppm
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:51.828792 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:03:51.829226 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42683/ using document root <none> and password file <none>
I20260924 05:03:51.829438 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:51.829480 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:51.829602 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:51.830690 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "b327e1571bbe45319315cbaf3aac19aa"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "9b0f289e8dfa7401dd6f48b8ec7b7c48"
server_key_iv: "5099c56e8526e940de38fbb8898e4ae4"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.831012 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b327e1571bbe45319315cbaf3aac19aa"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "9b0f289e8dfa7401dd6f48b8ec7b7c48"
server_key_iv: "5099c56e8526e940de38fbb8898e4ae4"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.832211 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:51.832847 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.832991 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:51.833067 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "b327e1571bbe45319315cbaf3aac19aa"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "9b0f289e8dfa7401dd6f48b8ec7b7c48"
server_key_iv: "5099c56e8526e940de38fbb8898e4ae4"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.833142 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:51.841530 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:51.843140 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:51.843256 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:51.843444 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:03:51.843701 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:51.843731 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.843750 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:51.843772 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.848834 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35259
I20260924 05:03:51.848889 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35259 every 8 connection(s)
I20260924 05:03:51.849153 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:03:51.854490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1158
I20260924 05:03:51.854568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:03:51.854712 392 external_mini_cluster.cc:1428] Setting key b12502b4a7d05e2bf7456292c6515662
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:51.857847 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.849752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38741 (local address 127.0.98.62:37359)
0924 05:03:51.849800 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:51.849802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:51.849998 (+ 196us) server_negotiation.cc:407] Connection header received
0924 05:03:51.850463 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:51.850465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:51.850477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:51.850526 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:51.851384 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.851884 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.852895 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.853109 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.853162 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.855263 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:51.855264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:51.855265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:51.855271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:51.856567 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.856962 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.856964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.856964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.857023 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.857235 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.857235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.857236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.857346 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:51.857401 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:51.857677 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 05:03:51.857785 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:03:51.858011 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.849742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37359 (local address 127.0.98.2:38741)
0924 05:03:51.849893 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:51.849925 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:51.850375 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:51.850590 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:51.850594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:51.851115 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:51.851289 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:51.851294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.851968 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.851969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:51.852820 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:51.852823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.853174 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.853176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:51.853286 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.853768 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:51.853776 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:51.855183 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:51.856657 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:51.856664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:51.856666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:51.856897 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:51.857075 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:51.857075 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:51.857076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:51.857180 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:51.857398 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:51.857403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:51.857548 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 05:03:51.857650 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 05:03:51.857817 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260924 05:03:51.858543 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37359
I20260924 05:03:51.858632 1287 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:51.858811 1287 heartbeater.cc:511] Master 127.0.98.62:37359 requested a full tablet report, sending...
I20260924 05:03:51.859144 962 ts_manager.cc:194] Registered new tserver with Master: b327e1571bbe45319315cbaf3aac19aa (127.0.98.2:35259)
I20260924 05:03:51.859489 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38741
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:03:51.875783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37359
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:51.951790 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:51.951967 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:51.951987 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:51.953112 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:51.953147 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:51.953173 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:03:51.954576 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37359
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:51.954783 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:51.954973 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:51.959322 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.959362 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:51.959334 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:51.959600 1292 server_base.cc:1034] running on GCE node
I20260924 05:03:51.959769 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:51.959964 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:51.961095 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790226231961083 us; error 26 us; skew 500 ppm
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:51.963195 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:03:51.963577 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43691/ using document root <none> and password file <none>
I20260924 05:03:51.963774 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:51.963820 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:51.963933 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:51.964838 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "30ac1a54e66040ea84378c0712a7c826"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "894a9dae70262b84d4e1742a4ecdfe2b"
server_key_iv: "7da6063b04b3f6be470b782933b9c7be"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.965140 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30ac1a54e66040ea84378c0712a7c826"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "894a9dae70262b84d4e1742a4ecdfe2b"
server_key_iv: "7da6063b04b3f6be470b782933b9c7be"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.966286 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:03:51.966863 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.966995 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:03:51.967060 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "30ac1a54e66040ea84378c0712a7c826"
format_stamp: "Formatted at 2026-09-24 05:03:51 on dist-test-slave-m01t"
server_key: "894a9dae70262b84d4e1742a4ecdfe2b"
server_key_iv: "7da6063b04b3f6be470b782933b9c7be"
server_key_version: "encryptionkey@0"
I20260924 05:03:51.967121 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:51.983613 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:51.985065 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:51.985183 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:51.985356 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:03:51.985595 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:51.985625 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.985651 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:51.985666 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:51.990830 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43887
I20260924 05:03:51.990885 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43887 every 8 connection(s)
I20260924 05:03:51.991135 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:03:51 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:51.998538 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.991751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34441 (local address 127.0.98.62:37359)
0924 05:03:51.991795 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:51.991797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:51.991958 (+ 161us) server_negotiation.cc:407] Connection header received
0924 05:03:51.992373 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:51.992374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:51.992382 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:51.992431 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:51.993221 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.993701 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.994615 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.994788 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.994831 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.996568 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:51.996570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:51.996570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:51.996577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:51.997609 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.997935 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.997935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.997936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.997983 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:51.998129 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:51.998129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:51.998129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:51.998216 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:51.998256 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:51.998425 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 05:03:51.998478 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:03:51.998690 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:51.991711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37359 (local address 127.0.98.3:34441)
0924 05:03:51.991846 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:51.991883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:51.992302 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:51.992480 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:51.992483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:51.992980 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:51.993155 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:51.993160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.993794 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.993795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:51.994556 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:51.994557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:51.994838 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:51.994838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:51.994924 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:51.995292 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:51.995299 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:51.996486 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:51.997668 (+ 1182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:51.997679 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:51.997680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:51.997880 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:51.998020 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:51.998020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:51.998020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:51.998096 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:51.998249 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:51.998251 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:51.998344 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 05:03:51.998411 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:03:51.998516 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260924 05:03:51.999097 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37359
I20260924 05:03:51.999178 1421 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:51.999341 1421 heartbeater.cc:511] Master 127.0.98.62:37359 requested a full tablet report, sending...
I20260924 05:03:51.999341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1292
I20260924 05:03:51.999421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:03:51.999571 392 external_mini_cluster.cc:1428] Setting key a360b7845a0c01aefecb5e0064e7d401
I20260924 05:03:51.999706 962 ts_manager.cc:194] Registered new tserver with Master: 30ac1a54e66040ea84378c0712a7c826 (127.0.98.3:43887)
I20260924 05:03:52.000193 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34441
I20260924 05:03:52.000324 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:03:52.006706 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.001769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.0.98.62:37359)
0924 05:03:52.001813 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.001815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.001863 (+ 48us) server_negotiation.cc:407] Connection header received
0924 05:03:52.001920 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.001921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.001929 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.001969 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:52.002475 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.002918 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.003606 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.003782 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.003832 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.004790 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:52.004791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:52.004792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:52.004798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:52.005760 (+ 962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.006033 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.006034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.006034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.006085 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.006237 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.006238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.006238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.006363 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:52.006444 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.006549 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.006627 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:03:52.008409 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:03:52.009858 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:03:52.015482 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.011597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43025 (local address 127.0.0.1:58612)
0924 05:03:52.011850 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:52.011860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:52.011909 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:52.012158 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:52.012159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:52.012161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:52.012203 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.012204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.013449 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.013450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.014331 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.014333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.015180 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.015180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.015249 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.015261 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 05:03:52.015365 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 05:03:52.015415 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":48,"thread_start_us":72,"threads_started":1}
I20260924 05:03:52.015702 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.011582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43887 (local address 127.0.0.1:50482)
0924 05:03:52.011684 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:52.011717 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:52.011769 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:52.012132 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:52.012134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:52.012136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:52.012191 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.012193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.013424 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.013424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.014674 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.014677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.015465 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.015466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.015523 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.015527 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:03:52.015558 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 05:03:52.015637 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":63,"thread_start_us":55,"threads_started":1}
I20260924 05:03:52.015805 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.011868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.0.98.2:35259)
0924 05:03:52.012237 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.012241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.012250 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:03:52.012283 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.012284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.012298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.012361 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:52.012455 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.013339 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.014872 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.015443 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.015498 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.015546 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.015697 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.015734 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":63,"threads_started":1}
I20260924 05:03:52.015880 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.011769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35259 (local address 127.0.0.1:34088)
0924 05:03:52.011871 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:52.011879 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:52.011919 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:52.012364 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:52.012365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:52.012367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:52.012397 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.012398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.013581 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.013582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.014813 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.014814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.015577 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.015578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.015618 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.015621 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:52.015665 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:03:52.015700 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":5,"thread_start_us":65,"threads_started":1}
I20260924 05:03:52.015990 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.011880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.0.98.3:43887)
0924 05:03:52.012009 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.012013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.012020 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:03:52.012043 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.012044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.012054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.012114 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:52.012250 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.013335 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.014743 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.015372 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.015800 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.015862 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.015887 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.015924 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260924 05:03:52.015502 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.011707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.0.98.1:43025)
0924 05:03:52.012027 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.012031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.012038 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:03:52.012067 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.012068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.012087 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.012150 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:52.012276 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.013358 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.014409 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.015102 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.015173 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.015224 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.015390 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.015429 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":59,"threads_started":1}
I20260924 05:03:52.016525 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 51a424564f3449a0993f6a1c6825ae0f (DEFAULT_TABLE table=test-table [id=c705fae3fa8b40b2833abfafc98906fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:52.016594 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 51a424564f3449a0993f6a1c6825ae0f (DEFAULT_TABLE table=test-table [id=c705fae3fa8b40b2833abfafc98906fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:52.016652 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 51a424564f3449a0993f6a1c6825ae0f (DEFAULT_TABLE table=test-table [id=c705fae3fa8b40b2833abfafc98906fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:52.016848 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a424564f3449a0993f6a1c6825ae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:52.016877 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a424564f3449a0993f6a1c6825ae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:52.016906 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a424564f3449a0993f6a1c6825ae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:52.019826 1437 tablet_bootstrap.cc:492] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2: Bootstrap starting.
I20260924 05:03:52.019826 1436 tablet_bootstrap.cc:492] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: Bootstrap starting.
I20260924 05:03:52.020074 1435 tablet_bootstrap.cc:492] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826: Bootstrap starting.
I20260924 05:03:52.020453 1437 tablet_bootstrap.cc:654] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:52.020453 1436 tablet_bootstrap.cc:654] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:52.020584 1435 tablet_bootstrap.cc:654] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:52.021051 1437 log.cc:824] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:52.021075 1435 log.cc:824] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:52.021135 1436 log.cc:824] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:52.021778 1436 tablet_bootstrap.cc:492] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: No bootstrap required, opened a new log
I20260924 05:03:52.021783 1435 tablet_bootstrap.cc:492] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826: No bootstrap required, opened a new log
I20260924 05:03:52.021843 1435 ts_tablet_manager.cc:1397] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:03:52.021844 1436 ts_tablet_manager.cc:1397] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:03:52.021895 1437 tablet_bootstrap.cc:492] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2: No bootstrap required, opened a new log
I20260924 05:03:52.021952 1437 ts_tablet_manager.cc:1397] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:03:52.023154 1436 raft_consensus.cc:348] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.023291 1436 raft_consensus.cc:374] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:52.023322 1436 raft_consensus.cc:729] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b327e1571bbe45319315cbaf3aac19aa, State: Initialized, Role: FOLLOWER
I20260924 05:03:52.023463 1436 consensus_queue.cc:260] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.023708 1436 ts_tablet_manager.cc:1428] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:03:52.023661 1435 raft_consensus.cc:348] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.023710 1437 raft_consensus.cc:348] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.023829 1287 heartbeater.cc:503] Master 127.0.98.62:37359 was elected leader, sending a full tablet report...
I20260924 05:03:52.023828 1435 raft_consensus.cc:374] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:52.023834 1437 raft_consensus.cc:374] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:52.023864 1435 raft_consensus.cc:729] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ac1a54e66040ea84378c0712a7c826, State: Initialized, Role: FOLLOWER
I20260924 05:03:52.023866 1437 raft_consensus.cc:729] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad35123159084357abc6fb54605307b2, State: Initialized, Role: FOLLOWER
I20260924 05:03:52.023969 1435 consensus_queue.cc:260] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.023969 1437 consensus_queue.cc:260] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.024220 1435 ts_tablet_manager.cc:1428] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:03:52.024219 1437 ts_tablet_manager.cc:1428] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:03:52.024343 1153 heartbeater.cc:503] Master 127.0.98.62:37359 was elected leader, sending a full tablet report...
I20260924 05:03:52.024562 1421 heartbeater.cc:503] Master 127.0.98.62:37359 was elected leader, sending a full tablet report...
I20260924 05:03:52.071857 1441 raft_consensus.cc:482] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:03:52.071971 1441 raft_consensus.cc:504] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.072301 1441 leader_election.cc:288] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ac1a54e66040ea84378c0712a7c826 (127.0.98.3:43887), ad35123159084357abc6fb54605307b2 (127.0.98.1:43025)
I20260924 05:03:52.075008 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.072545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38531 (local address 127.0.98.1:43025)
0924 05:03:52.072588 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.072591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.072771 (+ 180us) server_negotiation.cc:407] Connection header received
0924 05:03:52.072821 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.072822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.072833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.072878 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:52.073007 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.073448 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.074362 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.074724 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.074782 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.074791 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.074909 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.074945 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:03:52.075022 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.072461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43025 (local address 127.0.98.2:38531)
0924 05:03:52.072705 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:52.072714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:52.072765 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:52.072886 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:52.072886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:52.072888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:52.072950 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.072951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.073522 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.073523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.074279 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.074281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.074796 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.074797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.074853 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.074856 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:52.074900 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:03:52.074936 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":71,"thread_start_us":111,"threads_started":1}
I20260924 05:03:52.075273 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.072461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43887 (local address 127.0.98.2:59381)
0924 05:03:52.072519 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:52.072531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:52.072579 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:52.072701 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:52.072701 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:52.072703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:52.072764 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.072765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.073350 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.073351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.074119 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:52.074120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.074622 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.074623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:52.075117 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.075121 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:03:52.075174 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:03:52.075212 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:03:52.075407 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.072501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59381 (local address 127.0.98.3:43887)
0924 05:03:52.072552 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.072554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.072564 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:03:52.072630 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.072631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.072642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.072687 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:52.072818 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.073285 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.074206 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.074558 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.074615 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.074625 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.075298 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.075341 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:03:52.075385 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a424564f3449a0993f6a1c6825ae0f" candidate_uuid: "b327e1571bbe45319315cbaf3aac19aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad35123159084357abc6fb54605307b2" is_pre_election: true
I20260924 05:03:52.075528 1107 raft_consensus.cc:2454] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b327e1571bbe45319315cbaf3aac19aa in term 0.
I20260924 05:03:52.075686 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a424564f3449a0993f6a1c6825ae0f" candidate_uuid: "b327e1571bbe45319315cbaf3aac19aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ac1a54e66040ea84378c0712a7c826" is_pre_election: true
I20260924 05:03:52.075747 1176 leader_election.cc:302] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad35123159084357abc6fb54605307b2, b327e1571bbe45319315cbaf3aac19aa; no voters:
I20260924 05:03:52.075814 1375 raft_consensus.cc:2454] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b327e1571bbe45319315cbaf3aac19aa in term 0.
I20260924 05:03:52.075845 1441 raft_consensus.cc:2781] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:03:52.075902 1441 raft_consensus.cc:482] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:03:52.075925 1441 raft_consensus.cc:3037] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:52.076589 1441 raft_consensus.cc:504] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.076701 1441 leader_election.cc:288] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [CANDIDATE]: Term 1 election: Requested vote from peers 30ac1a54e66040ea84378c0712a7c826 (127.0.98.3:43887), ad35123159084357abc6fb54605307b2 (127.0.98.1:43025)
I20260924 05:03:52.076845 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a424564f3449a0993f6a1c6825ae0f" candidate_uuid: "b327e1571bbe45319315cbaf3aac19aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad35123159084357abc6fb54605307b2"
I20260924 05:03:52.076874 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a424564f3449a0993f6a1c6825ae0f" candidate_uuid: "b327e1571bbe45319315cbaf3aac19aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ac1a54e66040ea84378c0712a7c826"
I20260924 05:03:52.076916 1107 raft_consensus.cc:3037] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:52.076930 1375 raft_consensus.cc:3037] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:52.077549 1375 raft_consensus.cc:2454] T 51a424564f3449a0993f6a1c6825ae0f P 30ac1a54e66040ea84378c0712a7c826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b327e1571bbe45319315cbaf3aac19aa in term 1.
I20260924 05:03:52.077577 1107 raft_consensus.cc:2454] T 51a424564f3449a0993f6a1c6825ae0f P ad35123159084357abc6fb54605307b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b327e1571bbe45319315cbaf3aac19aa in term 1.
I20260924 05:03:52.077699 1176 leader_election.cc:302] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad35123159084357abc6fb54605307b2, b327e1571bbe45319315cbaf3aac19aa; no voters:
I20260924 05:03:52.077847 1441 raft_consensus.cc:2781] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:52.077996 1441 raft_consensus.cc:686] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [term 1 LEADER]: Becoming Leader. State: Replica: b327e1571bbe45319315cbaf3aac19aa, State: Running, Role: LEADER
I20260924 05:03:52.078074 1441 consensus_queue.cc:237] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } }
I20260924 05:03:52.078783 962 catalog_manager.cc:5495] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa reported cstate change: term changed from 0 to 1, leader changed from <none> to b327e1571bbe45319315cbaf3aac19aa (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b327e1571bbe45319315cbaf3aac19aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ac1a54e66040ea84378c0712a7c826" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad35123159084357abc6fb54605307b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b327e1571bbe45319315cbaf3aac19aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35259 } health_report { overall_health: HEALTHY } } }
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:03:52.091281 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.085133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.1:43025)
0924 05:03:52.085185 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.085187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.085281 (+ 94us) server_negotiation.cc:407] Connection header received
0924 05:03:52.085339 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.085340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.085349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.085392 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:52.085882 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.086310 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.086946 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.087125 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.087175 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.089197 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:52.089205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:52.089205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:52.089211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:52.090486 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.090770 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.090772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.090773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.090819 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.090939 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.090940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.090940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.091045 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:52.091135 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.091169 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.091221 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 05:03:52.099606 1288 tablet.cc:2369] T 51a424564f3449a0993f6a1c6825ae0f P b327e1571bbe45319315cbaf3aac19aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:03:52.110618 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.105094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.0.98.1:43025)
0924 05:03:52.105145 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.105147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.105203 (+ 56us) server_negotiation.cc:407] Connection header received
0924 05:03:52.105243 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.105244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.105251 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.105292 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:52.105768 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.106211 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.106837 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.107001 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.107040 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.108846 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:52.108847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:52.108847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:52.108853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:52.109848 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.110108 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.110109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.110109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.110150 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.110268 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.110268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.110268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.110341 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:52.110401 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.110498 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.110561 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:03:52.129035 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:52.123254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.0.98.1:43025)
0924 05:03:52.123307 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:52.123309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:52.123367 (+ 58us) server_negotiation.cc:407] Connection header received
0924 05:03:52.123436 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:52.123437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:52.123446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:52.123492 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:52.123959 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.124412 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.125078 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:52.125263 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:52.125315 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:52.127289 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:52.127290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:52.127290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:52.127296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:52.128297 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.128556 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.128556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.128557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.128595 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:52.128710 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:52.128710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:52.128710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:52.128788 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:52.128850 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:52.128926 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:03:52.128973 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 05:03:52.129434 1052 tablet_service.cc:2908] Rejecting scan request for tablet 51a424564f3449a0993f6a1c6825ae0f: Uninitialized: clean time has not yet been initialized
I20260924 05:03:52.130141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1024
I20260924 05:03:52.134397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1158
I20260924 05:03:52.138677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1292
I20260924 05:03:52.142367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 930
2026-09-24T05:03:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1761 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[1459](info): set up 2 sockets
Sep 24 05:03:52 dist-test-slave-m01t krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260924 05:03:53.111605 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 05:03:53.358934 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.247s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 05:03:53.594672 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.236s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 05:03:53.759714 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.165s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/test-user.keytab.
W20260924 05:03:54.765426 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.006s user 0.000s sys 0.001s
Sep 24 05:03:54 dist-test-slave-m01t krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 05:03:55.591796 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.808s user 0.000s sys 0.002s
I20260924 05:03:55.592129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43109
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:55.671834 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:55.672020 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:55.672040 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:55.673152 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:55.673192 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 05:03:55.673203 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:55.673226 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:55.673250 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:55.673271 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:55.674762 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:55.674968 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:55.675168 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:03:55.679457 1472 server_base.cc:1034] running on GCE node
W20260924 05:03:55.679428 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:55.679441 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:55.679432 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:55.679896 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:55.680150 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:55.681279 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790226235681267 us; error 28 us; skew 500 ppm
Sep 24 05:03:55 dist-test-slave-m01t krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:55.687042 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:55.687456 1472 webserver.cc:460] Webserver started at http://127.0.98.62:42253/ using document root <none> and password file <none>
I20260924 05:03:55.687654 1472 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:55.687700 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:55.687817 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:55.688787 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "d6170ce1d07c482ea768f907147a4225"
format_stamp: "Formatted at 2026-09-24 05:03:55 on dist-test-slave-m01t"
server_key: "17f1da37479daa352f810d59b9069335"
server_key_iv: "af64c33fa963525f4453160636509dd4"
server_key_version: "encryptionkey@0"
I20260924 05:03:55.689108 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6170ce1d07c482ea768f907147a4225"
format_stamp: "Formatted at 2026-09-24 05:03:55 on dist-test-slave-m01t"
server_key: "17f1da37479daa352f810d59b9069335"
server_key_iv: "af64c33fa963525f4453160636509dd4"
server_key_version: "encryptionkey@0"
I20260924 05:03:55.690351 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:03:55.690963 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:55.691112 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:55.691190 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "d6170ce1d07c482ea768f907147a4225"
format_stamp: "Formatted at 2026-09-24 05:03:55 on dist-test-slave-m01t"
server_key: "17f1da37479daa352f810d59b9069335"
server_key_iv: "af64c33fa963525f4453160636509dd4"
server_key_version: "encryptionkey@0"
I20260924 05:03:55.691250 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:55.715314 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:55.717897 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:55.718016 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:55.722342 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43109
I20260924 05:03:55.722391 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43109 every 8 connection(s)
I20260924 05:03:55.722704 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:55.723353 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:55.725687 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: Bootstrap starting.
I20260924 05:03:55.726130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1472
I20260924 05:03:55.726207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:55.726341 392 external_mini_cluster.cc:1428] Setting key 3ddbf01d6db7801f05ab2773932cb91f
I20260924 05:03:55.726337 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:55.726593 1550 log.cc:824] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:55.727415 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: No bootstrap required, opened a new log
I20260924 05:03:55.728734 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6170ce1d07c482ea768f907147a4225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43109 } }
I20260924 05:03:55.728868 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:55.728900 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6170ce1d07c482ea768f907147a4225, State: Initialized, Role: FOLLOWER
I20260924 05:03:55.729030 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6170ce1d07c482ea768f907147a4225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43109 } }
I20260924 05:03:55.729099 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:55.729130 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:55.729184 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 05:03:55 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:55.735677 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:55.726818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.0.98.62:43109)
0924 05:03:55.726979 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:55.726989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:55.727021 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:03:55.727471 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:55.727478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:55.727545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:55.727696 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:55.728274 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:55.729570 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:55.730352 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:55.730770 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:55.730865 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:55.732951 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:55.732956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:55.732956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:55.732969 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:55.734324 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:55.734634 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:55.734638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:55.734638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:55.734723 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:55.734896 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:55.734897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:55.734898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:55.735134 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:55.735257 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:55.735359 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 05:03:55.735493 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 05:03:55.784946 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6170ce1d07c482ea768f907147a4225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43109 } }
I20260924 05:03:55.785115 1550 leader_election.cc:302] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6170ce1d07c482ea768f907147a4225; no voters:
I20260924 05:03:55.785338 1550 leader_election.cc:288] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:55.785406 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:55.785585 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [term 1 LEADER]: Becoming Leader. State: Replica: d6170ce1d07c482ea768f907147a4225, State: Running, Role: LEADER
I20260924 05:03:55.785619 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:55.785786 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6170ce1d07c482ea768f907147a4225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43109 } }
I20260924 05:03:55.786183 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6170ce1d07c482ea768f907147a4225. Latest consensus state: current_term: 1 leader_uuid: "d6170ce1d07c482ea768f907147a4225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6170ce1d07c482ea768f907147a4225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43109 } } }
I20260924 05:03:55.786193 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6170ce1d07c482ea768f907147a4225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6170ce1d07c482ea768f907147a4225" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43109 } } }
I20260924 05:03:55.786348 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:55.786361 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:55.786612 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:55.787149 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:55.788745 1564 catalog_manager.cc:1347] Generated new cluster ID: 8501fcf299c8429990ae7ce38d44ce00
I20260924 05:03:55.788800 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:55.797441 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:55.798049 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:55.802763 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: Generated new TSK 0
I20260924 05:03:55.802909 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260924 05:03:56.577477 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.772s user 0.000s sys 0.003s
I20260924 05:03:56.577805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43109
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:56.654696 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:56.654884 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:56.654906 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:56.655993 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:56.656036 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:56.656059 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:03:56.656131 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:03:56.657820 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:56.657994 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:56.658164 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:56.663360 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:56.663378 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:56.663486 1576 server_base.cc:1034] running on GCE node
W20260924 05:03:56.663655 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:56.663836 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:56.664036 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:56.665169 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790226236665151 us; error 30 us; skew 500 ppm
Sep 24 05:03:56 dist-test-slave-m01t krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:56.667529 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:03:56.667872 1576 webserver.cc:460] Webserver started at http://127.0.98.1:41073/ using document root <none> and password file <none>
I20260924 05:03:56.668040 1576 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:56.668075 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:56.668175 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:56.669095 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "b1e3c2055da640b4beca68229aa854d7"
format_stamp: "Formatted at 2026-09-24 05:03:56 on dist-test-slave-m01t"
server_key: "de9977425f24f861886f3bf3aa3a0fda"
server_key_iv: "5889a759cd9c4aedc1e1f751203431a2"
server_key_version: "encryptionkey@0"
I20260924 05:03:56.669379 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b1e3c2055da640b4beca68229aa854d7"
format_stamp: "Formatted at 2026-09-24 05:03:56 on dist-test-slave-m01t"
server_key: "de9977425f24f861886f3bf3aa3a0fda"
server_key_iv: "5889a759cd9c4aedc1e1f751203431a2"
server_key_version: "encryptionkey@0"
I20260924 05:03:56.670522 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 05:03:56.671089 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:56.671233 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:56.671303 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "b1e3c2055da640b4beca68229aa854d7"
format_stamp: "Formatted at 2026-09-24 05:03:56 on dist-test-slave-m01t"
server_key: "de9977425f24f861886f3bf3aa3a0fda"
server_key_iv: "5889a759cd9c4aedc1e1f751203431a2"
server_key_version: "encryptionkey@0"
I20260924 05:03:56.671365 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:56.684337 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:56.685966 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:56.686079 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:56.686506 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:56.686551 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:56.686573 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:56.686594 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:56.693037 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33053
I20260924 05:03:56.693169 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33053 every 8 connection(s)
Sep 24 05:03:56 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:56.693441 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:03:56.696133 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:56.686899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33163 (local address 127.0.98.62:43109)
0924 05:03:56.687643 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:56.687645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:56.687653 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:56.687679 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:56.687680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:56.687692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:56.687753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:56.688589 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.689359 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.690896 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.691210 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.691284 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:56.693830 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:56.693832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:56.693833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:56.693840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:56.695060 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:56.695400 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:56.695401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:56.695402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:56.695446 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:56.695615 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:56.695616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:56.695616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:56.695739 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:56.695789 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:56.696006 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 05:03:56.696066 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":58,"threads_started":1}
I20260924 05:03:56.696290 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:56.686870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.98.1:33163)
0924 05:03:56.687041 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:56.687079 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:56.687507 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:56.687861 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:56.687866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:56.688212 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:56.688482 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:56.688489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.689448 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.689449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:56.690802 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:56.690805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.691288 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.691288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:56.691432 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:56.691974 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:56.691984 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:56.693701 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:56.695125 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:56.695130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:56.695132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:56.695341 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:56.695489 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:56.695489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:56.695489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:56.695573 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:56.695784 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:56.695787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:56.695907 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 05:03:56.695982 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:03:56.696100 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260924 05:03:56.696717 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43109
I20260924 05:03:56.696789 1708 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:56.696964 1708 heartbeater.cc:511] Master 127.0.98.62:43109 requested a full tablet report, sending...
I20260924 05:03:56.697522 1503 ts_manager.cc:194] Registered new tserver with Master: b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053)
I20260924 05:03:56.698313 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33163
I20260924 05:03:56.701834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1576
I20260924 05:03:56.701922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:03:56.702102 392 external_mini_cluster.cc:1428] Setting key f4b35d68750ed24ba24511d9801025f0
I20260924 05:03:56.702438 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:56.697690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55487 (local address 127.0.98.62:43109)
0924 05:03:56.697752 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:56.697754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:56.697816 (+ 62us) server_negotiation.cc:407] Connection header received
0924 05:03:56.697860 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:56.697861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:56.697870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:56.697931 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:56.698175 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.698742 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.699472 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.699643 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.699683 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:56.700424 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:56.700425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:56.700425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:56.700429 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:56.701397 (+ 968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:56.701649 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:56.701650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:56.701650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:56.701690 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:56.701956 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:56.701957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:56.701958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:56.702090 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:56.702161 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:56.702305 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:03:56.702383 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:03:56.702821 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:56.697650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.98.1:55487)
0924 05:03:56.697767 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:56.697776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:56.697826 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:56.697934 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:56.697934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:56.698060 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:56.698113 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:56.698114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.698818 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.698818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:56.699412 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:56.699414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:56.699690 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:56.699690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:56.699734 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:56.700103 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:56.700107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:56.700368 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:56.701452 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:56.701453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:56.701453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:56.701599 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:56.701827 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:56.701828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:56.701829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:56.701893 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:56.702158 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:56.702158 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:56.702200 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 05:03:56.702272 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:03:56.702349 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":6,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 05:03:57.308915 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.606s user 0.000s sys 0.002s
I20260924 05:03:57.309254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43109
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:57.387158 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:57.387341 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:57.387362 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:57.388430 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:57.388465 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:57.388484 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:03:57.388499 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:03:57.389936 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:57.390136 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:57.390311 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:57.395063 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:57.395021 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:57.395005 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:57.395000 1717 server_base.cc:1034] running on GCE node
I20260924 05:03:57.395401 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:57.395627 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:57.396746 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790226237396741 us; error 31 us; skew 500 ppm
Sep 24 05:03:57 dist-test-slave-m01t krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:57.398999 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:03:57.399453 1717 webserver.cc:460] Webserver started at http://127.0.98.2:42713/ using document root <none> and password file <none>
I20260924 05:03:57.399689 1717 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:57.399744 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:57.399859 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:57.400861 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "0566a286e4d5498b95e331c9dd9201d6"
format_stamp: "Formatted at 2026-09-24 05:03:57 on dist-test-slave-m01t"
server_key: "da081d722e5798444abbbcaafd162b15"
server_key_iv: "16e6eade5b76951c3c41d57c0173af6e"
server_key_version: "encryptionkey@0"
I20260924 05:03:57.401175 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0566a286e4d5498b95e331c9dd9201d6"
format_stamp: "Formatted at 2026-09-24 05:03:57 on dist-test-slave-m01t"
server_key: "da081d722e5798444abbbcaafd162b15"
server_key_iv: "16e6eade5b76951c3c41d57c0173af6e"
server_key_version: "encryptionkey@0"
I20260924 05:03:57.402402 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:03:57.403021 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:57.403169 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:03:57.403245 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "0566a286e4d5498b95e331c9dd9201d6"
format_stamp: "Formatted at 2026-09-24 05:03:57 on dist-test-slave-m01t"
server_key: "da081d722e5798444abbbcaafd162b15"
server_key_iv: "16e6eade5b76951c3c41d57c0173af6e"
server_key_version: "encryptionkey@0"
I20260924 05:03:57.403296 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:57.426188 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:57.427981 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:57.428102 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:57.428571 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:57.428612 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:57.428637 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:57.428647 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:57.434908 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42917
I20260924 05:03:57.434990 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42917 every 8 connection(s)
I20260924 05:03:57.435312 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:03:57 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:57.438391 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:57.429067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57413 (local address 127.0.98.62:43109)
0924 05:03:57.429174 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:57.429177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:57.429342 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:03:57.429926 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:57.429928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:57.429944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:57.430024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:57.430861 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.431640 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.433156 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.433456 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.433538 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:57.435919 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:57.435921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:57.435921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:57.435928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:57.437279 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:57.437605 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:57.437606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:57.437606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:57.437655 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:57.437886 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:57.437887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:57.437887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:57.437990 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:57.438039 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:57.438260 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 05:03:57.438322 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
I20260924 05:03:57.438542 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:57.429076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.98.2:57413)
0924 05:03:57.429228 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:57.429266 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:57.429806 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:57.430113 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:57.430118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:57.430471 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:57.430753 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:57.430761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.431746 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.431747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:57.433055 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:57.433058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.433541 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.433542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:57.433657 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:57.434270 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:57.434281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:57.435835 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:57.437342 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:57.437346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:57.437347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:57.437549 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:57.437739 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:57.437740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:57.437740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:57.437838 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:57.438035 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:57.438037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:57.438157 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 05:03:57.438229 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:03:57.438350 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260924 05:03:57.439009 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43109
I20260924 05:03:57.439092 1849 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:57.439256 1849 heartbeater.cc:511] Master 127.0.98.62:43109 requested a full tablet report, sending...
I20260924 05:03:57.439604 1503 ts_manager.cc:194] Registered new tserver with Master: 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917)
I20260924 05:03:57.439973 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57413
I20260924 05:03:57.443612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1717
I20260924 05:03:57.443701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:03:57.443873 392 external_mini_cluster.cc:1428] Setting key f0223758047db26e60919680d73c013f
I20260924 05:03:57.444756 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:57.439960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56897 (local address 127.0.98.62:43109)
0924 05:03:57.440081 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:57.440082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:57.440090 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:57.440106 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:57.440107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:57.440115 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:57.440161 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:57.440475 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.441008 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.441622 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.441860 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.441904 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:57.442580 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:57.442581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:57.442581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:57.442586 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:57.443739 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:57.444059 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:57.444060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:57.444060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:57.444117 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:57.444284 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:57.444285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:57.444285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:57.444446 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:57.444509 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:57.444639 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 05:03:57.444700 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 05:03:57.444797 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:57.439898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.98.2:56897)
0924 05:03:57.440001 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:57.440010 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:57.440055 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:57.440165 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:57.440165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:57.440338 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:57.440394 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:57.440395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.441081 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.441082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:57.441562 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:57.441563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:57.441918 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:57.441919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:57.441965 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:57.442277 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:57.442280 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:57.442512 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:57.443814 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:57.443815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:57.443816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:57.443998 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:57.444180 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:57.444181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:57.444181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:57.444237 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:57.444507 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:57.444507 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:57.444548 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:03:57.444621 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 05:03:57.444709 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 05:03:57.699209 1708 heartbeater.cc:503] Master 127.0.98.62:43109 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 05:03:58.181190 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.737s user 0.000s sys 0.003s
I20260924 05:03:58.181484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43109
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:58.258212 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:58.258407 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:58.258437 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:58.259553 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:58.259593 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:58.259619 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:03:58.259640 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:03:58.261111 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:58.261358 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:58.261548 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:58.265929 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:58.265951 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:58.265929 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:58.266425 1858 server_base.cc:1034] running on GCE node
I20260924 05:03:58.266590 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:58.266767 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:58.267890 1858 hybrid_clock.cc:648] HybridClock initialized: now 1790226238267876 us; error 27 us; skew 500 ppm
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:58.270056 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:03:58.270453 1858 webserver.cc:460] Webserver started at http://127.0.98.3:41263/ using document root <none> and password file <none>
I20260924 05:03:58.270645 1858 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:58.270687 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:58.270803 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:58.271757 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "c07d4335ac004b61ae99f476c47ca5c7"
format_stamp: "Formatted at 2026-09-24 05:03:58 on dist-test-slave-m01t"
server_key: "db75f069a337fa219c65c7b61b67f41e"
server_key_iv: "0ea296b9e28cc27e49d758f964109d3f"
server_key_version: "encryptionkey@0"
I20260924 05:03:58.272063 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c07d4335ac004b61ae99f476c47ca5c7"
format_stamp: "Formatted at 2026-09-24 05:03:58 on dist-test-slave-m01t"
server_key: "db75f069a337fa219c65c7b61b67f41e"
server_key_iv: "0ea296b9e28cc27e49d758f964109d3f"
server_key_version: "encryptionkey@0"
I20260924 05:03:58.273193 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:58.273825 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.273950 1858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:58.274026 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "c07d4335ac004b61ae99f476c47ca5c7"
format_stamp: "Formatted at 2026-09-24 05:03:58 on dist-test-slave-m01t"
server_key: "db75f069a337fa219c65c7b61b67f41e"
server_key_iv: "0ea296b9e28cc27e49d758f964109d3f"
server_key_version: "encryptionkey@0"
I20260924 05:03:58.274089 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:58.284585 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:58.286563 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:58.286679 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:58.287144 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:03:58.287189 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.287215 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:03:58.287232 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:58.293489 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46791
I20260924 05:03:58.293565 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46791 every 8 connection(s)
I20260924 05:03:58.293965 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:03:58.295624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 1858
I20260924 05:03:58.295722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:03:58.295897 392 external_mini_cluster.cc:1428] Setting key f15fda43891dd00bb64fed9c314dde34
I20260924 05:03:58.296463 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.287501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54385 (local address 127.0.98.62:43109)
0924 05:03:58.287633 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.287635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.287760 (+ 125us) server_negotiation.cc:407] Connection header received
0924 05:03:58.288201 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.288202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.288217 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.288302 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:58.288959 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.289744 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.291195 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.291501 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.291581 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.293743 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:58.293744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:58.293745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:58.293753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:58.295182 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.295519 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.295520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.295521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.295566 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.295813 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.295814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.295815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.295923 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:58.295979 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.296292 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.296370 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260924 05:03:58.296602 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.287483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.98.3:54385)
0924 05:03:58.287647 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.287683 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.288116 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.288349 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.288352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.288611 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:58.288859 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.288865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.289833 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.289834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.291091 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.291094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.291614 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.291615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.291756 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.292266 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:58.292274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:58.293628 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:58.295255 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:58.295260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:58.295262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:58.295460 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:58.295617 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:58.295618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:58.295618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:58.295747 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:58.295983 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:58.295989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:58.296127 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.296230 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.296377 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260924 05:03:58.297003 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43109
I20260924 05:03:58.297089 1990 heartbeater.cc:464] Registering TS with master...
I20260924 05:03:58.297246 1990 heartbeater.cc:511] Master 127.0.98.62:43109 requested a full tablet report, sending...
I20260924 05:03:58.297554 1504 ts_manager.cc:194] Registered new tserver with Master: c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791)
I20260924 05:03:58.298044 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54385
I20260924 05:03:58.303102 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.297940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43723 (local address 127.0.98.62:43109)
0924 05:03:58.297978 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.297979 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.298032 (+ 53us) server_negotiation.cc:407] Connection header received
0924 05:03:58.298096 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.298097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.298106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.298151 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:58.298537 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.299174 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.300020 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.300322 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.300385 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.301334 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:58.301335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:58.301335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:58.301342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:58.302373 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.302642 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.302642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.302643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.302686 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.302790 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.302790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.302790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.302867 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:58.302905 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.302993 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.303046 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:03:58.303115 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.297871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.98.3:43723)
0924 05:03:58.297980 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.297989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.298045 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.298176 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.298177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.298371 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:58.298449 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.298450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.299260 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.299261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.299940 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.299942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.300386 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.300387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.300480 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.300866 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:58.300871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:58.301238 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:58.302434 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:58.302435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:58.302435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:58.302587 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:58.302718 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:58.302719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:58.302719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:58.302759 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:58.302897 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:58.302897 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:58.302927 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.302978 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.303042 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260924 05:03:58.307291 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:58.326651 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.320767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.0.98.62:43109)
0924 05:03:58.320811 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.320813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.320820 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:03:58.320885 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.320886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.320894 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.320932 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:58.321396 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.321912 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.322578 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.322760 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.322801 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.324745 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:58.324747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:58.324747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:58.324753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:58.325846 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.326124 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.326124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.326124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.326171 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.326293 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.326293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.326294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.326388 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:58.326459 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.326531 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.326585 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:03:58.328100 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:03:58.329619 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:03:58.335731 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.331868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33053 (local address 127.0.0.1:41586)
0924 05:03:58.332072 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.332109 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.332176 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.332549 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.332550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.332552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.332602 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.332603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.333516 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.333517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.334743 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.334745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.335441 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.335442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.335514 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.335527 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.335592 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.335641 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260924 05:03:58.335803 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.332123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.0.98.3:46791)
0924 05:03:58.332292 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.332298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.332308 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:03:58.332370 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.332372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.332387 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.332459 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.332744 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.333822 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.334807 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.335491 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.335562 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.335644 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.335671 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.335707 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260924 05:03:58.336009 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.331868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42917 (local address 127.0.0.1:56066)
0924 05:03:58.332067 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.332109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.332175 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.332553 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.332554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.332555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.332603 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.332603 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.333856 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.333857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.334914 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.334916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.335812 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.335813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.335865 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.335868 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.335919 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.335951 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":30,"thread_start_us":92,"threads_started":1}
I20260924 05:03:58.335742 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.331870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46791 (local address 127.0.0.1:44210)
0924 05:03:58.332269 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.332278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.332312 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.332471 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.332474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.332477 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.332571 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.332573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.333911 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.333912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.334722 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.334724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.335559 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.335560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.335610 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.335613 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.335647 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.335685 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260924 05:03:58.336248 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.332309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.0.98.2:42917)
0924 05:03:58.332419 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.332423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.332432 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:03:58.332466 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.332468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.332482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.332559 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.332659 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.333780 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.334991 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.335689 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.335781 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.335834 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.336130 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.336175 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260924 05:03:58.336460 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.332099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.0.98.1:33053)
0924 05:03:58.332403 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.332408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.332417 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:03:58.332447 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.332449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.332462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.332669 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.332723 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.333440 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.334834 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.335370 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.335433 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.335494 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.336332 (+ 838us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.336381 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":50,"threads_started":1}
I20260924 05:03:58.336743 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a552468e749c4534922c1e65ca8a206f (DEFAULT_TABLE table=test-table [id=80b0bf01ffec4f8e9b4354cdc0243a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:58.337148 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a552468e749c4534922c1e65ca8a206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.337091 1783 tablet_service.cc:1467] Processing CreateTablet for tablet a552468e749c4534922c1e65ca8a206f (DEFAULT_TABLE table=test-table [id=80b0bf01ffec4f8e9b4354cdc0243a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:58.337389 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a552468e749c4534922c1e65ca8a206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.339777 2014 tablet_bootstrap.cc:492] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Bootstrap starting.
I20260924 05:03:58.340132 2015 tablet_bootstrap.cc:492] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Bootstrap starting.
I20260924 05:03:58.340485 2014 tablet_bootstrap.cc:654] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.340672 2015 tablet_bootstrap.cc:654] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.340608 1642 tablet_service.cc:1467] Processing CreateTablet for tablet a552468e749c4534922c1e65ca8a206f (DEFAULT_TABLE table=test-table [id=80b0bf01ffec4f8e9b4354cdc0243a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:58.340806 2014 log.cc:824] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:58.340934 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a552468e749c4534922c1e65ca8a206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.340947 2015 log.cc:824] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:58.341549 2015 tablet_bootstrap.cc:492] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: No bootstrap required, opened a new log
I20260924 05:03:58.341604 2015 ts_tablet_manager.cc:1397] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:03:58.342342 2014 tablet_bootstrap.cc:492] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: No bootstrap required, opened a new log
I20260924 05:03:58.342406 2014 ts_tablet_manager.cc:1397] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:03:58.343475 2018 tablet_bootstrap.cc:492] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Bootstrap starting.
I20260924 05:03:58.343461 2015 raft_consensus.cc:348] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.343605 2015 raft_consensus.cc:374] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.343636 2015 raft_consensus.cc:729] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07d4335ac004b61ae99f476c47ca5c7, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.343750 2015 consensus_queue.cc:260] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.343928 2018 tablet_bootstrap.cc:654] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.344023 1990 heartbeater.cc:503] Master 127.0.98.62:43109 was elected leader, sending a full tablet report...
I20260924 05:03:58.343991 2015 ts_tablet_manager.cc:1428] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:03:58.344218 2018 log.cc:824] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:58.344727 2014 raft_consensus.cc:348] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.344862 2014 raft_consensus.cc:374] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.344892 2014 raft_consensus.cc:729] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0566a286e4d5498b95e331c9dd9201d6, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.344936 2018 tablet_bootstrap.cc:492] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: No bootstrap required, opened a new log
I20260924 05:03:58.344985 2018 ts_tablet_manager.cc:1397] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:03:58.344991 2014 consensus_queue.cc:260] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.345196 2014 ts_tablet_manager.cc:1428] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:03:58.345227 1849 heartbeater.cc:503] Master 127.0.98.62:43109 was elected leader, sending a full tablet report...
I20260924 05:03:58.346863 2018 raft_consensus.cc:348] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.346975 2018 raft_consensus.cc:374] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.347004 2018 raft_consensus.cc:729] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e3c2055da640b4beca68229aa854d7, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.347103 2018 consensus_queue.cc:260] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.347286 2018 ts_tablet_manager.cc:1428] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 05:03:58.436179 1850 tablet.cc:2369] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:03:58.444686 1709 tablet.cc:2369] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:03:58.539481 2019 raft_consensus.cc:482] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:03:58.539594 2019 raft_consensus.cc:504] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.539958 2019 leader_election.cc:288] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053), 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917)
I20260924 05:03:58.543576 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.540192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60031 (local address 127.0.98.2:42917)
0924 05:03:58.540236 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.540238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.540249 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:03:58.540312 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.540313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.540323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.540380 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.540534 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.541282 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.542711 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.543244 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.543311 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.543320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.543476 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.543508 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:03:58.543576 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.540227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40181 (local address 127.0.98.1:33053)
0924 05:03:58.540267 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.540268 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.540432 (+ 164us) server_negotiation.cc:407] Connection header received
0924 05:03:58.540484 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.540485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.540494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.540540 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.540651 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.541335 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.542705 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.543261 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.543311 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.543320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.543474 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.543509 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:03:58.543615 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.540145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42917 (local address 127.0.98.3:60031)
0924 05:03:58.540197 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.540208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.540253 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.540393 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.540394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.540396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.540475 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.540477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.541353 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.541353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.542631 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.542634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.543322 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.543323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.543414 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.543418 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.543464 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.543505 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":61}
I20260924 05:03:58.543615 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.540155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33053 (local address 127.0.98.3:40181)
0924 05:03:58.540362 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.540371 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.540422 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.540543 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.540543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.540545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.540600 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.540601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.541396 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.541396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.542631 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.542634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.543324 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.543324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.543414 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.543418 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.543464 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.543505 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260924 05:03:58.544031 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a552468e749c4534922c1e65ca8a206f" candidate_uuid: "c07d4335ac004b61ae99f476c47ca5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0566a286e4d5498b95e331c9dd9201d6" is_pre_election: true
I20260924 05:03:58.544036 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a552468e749c4534922c1e65ca8a206f" candidate_uuid: "c07d4335ac004b61ae99f476c47ca5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e3c2055da640b4beca68229aa854d7" is_pre_election: true
I20260924 05:03:58.544162 1662 raft_consensus.cc:2454] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c07d4335ac004b61ae99f476c47ca5c7 in term 0.
I20260924 05:03:58.544174 1803 raft_consensus.cc:2454] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c07d4335ac004b61ae99f476c47ca5c7 in term 0.
I20260924 05:03:58.544376 1877 leader_election.cc:302] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0566a286e4d5498b95e331c9dd9201d6, b1e3c2055da640b4beca68229aa854d7, c07d4335ac004b61ae99f476c47ca5c7; no voters:
I20260924 05:03:58.544535 2019 raft_consensus.cc:2781] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:03:58.544574 2019 raft_consensus.cc:482] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:03:58.544588 2019 raft_consensus.cc:3037] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Advancing to term 1
W20260924 05:03:58.545014 1991 tablet.cc:2369] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:03:58.545375 2019 raft_consensus.cc:504] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.545482 2019 leader_election.cc:288] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [CANDIDATE]: Term 1 election: Requested vote from peers b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053), 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917)
I20260924 05:03:58.545617 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a552468e749c4534922c1e65ca8a206f" candidate_uuid: "c07d4335ac004b61ae99f476c47ca5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e3c2055da640b4beca68229aa854d7"
I20260924 05:03:58.545692 1662 raft_consensus.cc:3037] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.545778 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a552468e749c4534922c1e65ca8a206f" candidate_uuid: "c07d4335ac004b61ae99f476c47ca5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0566a286e4d5498b95e331c9dd9201d6"
I20260924 05:03:58.545849 1803 raft_consensus.cc:3037] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.546373 1662 raft_consensus.cc:2454] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07d4335ac004b61ae99f476c47ca5c7 in term 1.
I20260924 05:03:58.546450 1803 raft_consensus.cc:2454] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07d4335ac004b61ae99f476c47ca5c7 in term 1.
I20260924 05:03:58.546535 1877 leader_election.cc:302] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e3c2055da640b4beca68229aa854d7, c07d4335ac004b61ae99f476c47ca5c7; no voters:
I20260924 05:03:58.546610 2019 raft_consensus.cc:2781] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:58.546737 2019 raft_consensus.cc:686] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 1 LEADER]: Becoming Leader. State: Replica: c07d4335ac004b61ae99f476c47ca5c7, State: Running, Role: LEADER
I20260924 05:03:58.546813 2019 consensus_queue.cc:237] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } }
I20260924 05:03:58.547560 1504 catalog_manager.cc:5495] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c07d4335ac004b61ae99f476c47ca5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } health_report { overall_health: UNKNOWN } } }
I20260924 05:03:58.559685 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.557424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.0.98.3:46791)
0924 05:03:58.557461 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.557462 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.557486 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:03:58.557568 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.557569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.557587 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.557640 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:58.557765 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.558214 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.558947 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.559153 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.559204 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.559383 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.559478 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.559543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.559603 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.559622 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:03:58.561142 1803 raft_consensus.cc:1261] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Refusing update from remote peer c07d4335ac004b61ae99f476c47ca5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:58.561142 1662 raft_consensus.cc:1261] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Refusing update from remote peer c07d4335ac004b61ae99f476c47ca5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:58.561445 2024 consensus_queue.cc:1035] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:58.561546 2019 consensus_queue.cc:1035] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:58.564180 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332766673144238080 to 7332766673088221184. Current Snapshot: MvccSnapshot[applied={T|T < 7332766673144238080 or (T in {7332766673144238080})}]
I20260924 05:03:58.564261 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332766673144238080 to 7332766673088221184. Current Snapshot: MvccSnapshot[applied={T|T < 7332766673144238080}]
I20260924 05:03:58.565007 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332766673144238080 to 7332766673088221184. Current Snapshot: MvccSnapshot[applied={T|T < 7332766673144238080 or (T in {7332766673144238080})}]
I20260924 05:03:58.567754 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:03:58.567831 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:03:58.568427 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: Sending DeleteTablet for 3 replicas of tablet a552468e749c4534922c1e65ca8a206f
I20260924 05:03:58.568704 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a552468e749c4534922c1e65ca8a206f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44210
I20260924 05:03:58.568718 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet a552468e749c4534922c1e65ca8a206f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41586
I20260924 05:03:58.568898 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet a552468e749c4534922c1e65ca8a206f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56066
I20260924 05:03:58.569077 2038 tablet_replica.cc:330] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: stopping tablet replica
I20260924 05:03:58.569137 2037 tablet_replica.cc:330] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: stopping tablet replica
I20260924 05:03:58.569198 2038 raft_consensus.cc:2229] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:03:58.569248 2037 raft_consensus.cc:2229] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:03:58.569309 2038 raft_consensus.cc:2258] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:03:58.569360 2037 raft_consensus.cc:2258] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:03:58.569355 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:03:58.569485 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:03:58.569514 2036 tablet_replica.cc:330] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: stopping tablet replica
I20260924 05:03:58.569581 2036 raft_consensus.cc:2229] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:03:58.569671 2036 raft_consensus.cc:2258] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:03:58.569731 2038 ts_tablet_manager.cc:1905] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:03:58.570084 2036 ts_tablet_manager.cc:1905] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:03:58.570842 1642 tablet_service.cc:1467] Processing CreateTablet for tablet a6517e3a26ac45d88d02663b42dd04b9 (DEFAULT_TABLE table=test-table [id=e503da520d0940a5b116019362311b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:58.570945 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6517e3a26ac45d88d02663b42dd04b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.571290 2038 ts_tablet_manager.cc:1918] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:03:58.571347 2038 log.cc:1196] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/a552468e749c4534922c1e65ca8a206f
I20260924 05:03:58.571574 2038 ts_tablet_manager.cc:1939] T a552468e749c4534922c1e65ca8a206f P 0566a286e4d5498b95e331c9dd9201d6: Deleting consensus metadata
I20260924 05:03:58.571560 2036 ts_tablet_manager.cc:1918] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:03:58.571630 2036 log.cc:1196] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/a552468e749c4534922c1e65ca8a206f
I20260924 05:03:58.571849 2036 ts_tablet_manager.cc:1939] T a552468e749c4534922c1e65ca8a206f P c07d4335ac004b61ae99f476c47ca5c7: Deleting consensus metadata
I20260924 05:03:58.572156 2018 tablet_bootstrap.cc:492] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Bootstrap starting.
I20260924 05:03:58.572325 1489 catalog_manager.cc:4841] TS c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791): tablet a552468e749c4534922c1e65ca8a206f (table test-table [id=80b0bf01ffec4f8e9b4354cdc0243a53]) successfully deleted
I20260924 05:03:58.572480 1490 catalog_manager.cc:4841] TS 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917): tablet a552468e749c4534922c1e65ca8a206f (table test-table [id=80b0bf01ffec4f8e9b4354cdc0243a53]) successfully deleted
I20260924 05:03:58.572539 2018 tablet_bootstrap.cc:654] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.572674 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a6517e3a26ac45d88d02663b42dd04b9 (DEFAULT_TABLE table=test-table [id=e503da520d0940a5b116019362311b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:58.572751 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6517e3a26ac45d88d02663b42dd04b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.572829 1783 tablet_service.cc:1467] Processing CreateTablet for tablet a6517e3a26ac45d88d02663b42dd04b9 (DEFAULT_TABLE table=test-table [id=e503da520d0940a5b116019362311b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:03:58.572904 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6517e3a26ac45d88d02663b42dd04b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.574131 2014 tablet_bootstrap.cc:492] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Bootstrap starting.
I20260924 05:03:58.574291 2018 tablet_bootstrap.cc:492] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: No bootstrap required, opened a new log
I20260924 05:03:58.574333 2018 ts_tablet_manager.cc:1397] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:03:58.574469 2018 raft_consensus.cc:348] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.574517 2014 tablet_bootstrap.cc:654] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.574527 2018 raft_consensus.cc:374] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.574546 2018 raft_consensus.cc:729] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e3c2055da640b4beca68229aa854d7, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.574579 2018 consensus_queue.cc:260] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.574663 2015 tablet_bootstrap.cc:492] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Bootstrap starting.
I20260924 05:03:58.574674 2018 ts_tablet_manager.cc:1428] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.575474 2015 tablet_bootstrap.cc:654] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.576098 2037 ts_tablet_manager.cc:1905] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:03:58.577606 2015 tablet_bootstrap.cc:492] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: No bootstrap required, opened a new log
I20260924 05:03:58.577654 2015 ts_tablet_manager.cc:1397] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:03:58.577739 2037 ts_tablet_manager.cc:1918] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:03:58.577798 2037 log.cc:1196] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/a552468e749c4534922c1e65ca8a206f
I20260924 05:03:58.577852 2015 raft_consensus.cc:348] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.577910 2015 raft_consensus.cc:374] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.577929 2015 raft_consensus.cc:729] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07d4335ac004b61ae99f476c47ca5c7, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.577976 2015 consensus_queue.cc:260] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.578044 2037 ts_tablet_manager.cc:1939] T a552468e749c4534922c1e65ca8a206f P b1e3c2055da640b4beca68229aa854d7: Deleting consensus metadata
I20260924 05:03:58.578055 2015 ts_tablet_manager.cc:1428] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.578373 1491 catalog_manager.cc:4841] TS b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053): tablet a552468e749c4534922c1e65ca8a206f (table test-table [id=80b0bf01ffec4f8e9b4354cdc0243a53]) successfully deleted
I20260924 05:03:58.578629 2014 tablet_bootstrap.cc:492] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: No bootstrap required, opened a new log
I20260924 05:03:58.578671 2014 ts_tablet_manager.cc:1397] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 05:03:58.578864 2014 raft_consensus.cc:348] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.578925 2014 raft_consensus.cc:374] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.578948 2014 raft_consensus.cc:729] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0566a286e4d5498b95e331c9dd9201d6, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.578989 2014 consensus_queue.cc:260] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.579147 2014 ts_tablet_manager.cc:1428] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.710207 2022 raft_consensus.cc:482] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:03:58.710295 2022 raft_consensus.cc:504] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.710598 2022 leader_election.cc:288] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917), c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791)
I20260924 05:03:58.713317 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.710869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53185 (local address 127.0.98.2:42917)
0924 05:03:58.710913 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.710914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.710948 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:03:58.711028 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.711029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.711040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.711088 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.711218 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.711684 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.712659 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.713024 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.713079 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.713089 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.713225 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.713256 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:03:58.713337 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.710866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41541 (local address 127.0.98.3:46791)
0924 05:03:58.710919 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.710921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.711012 (+ 91us) server_negotiation.cc:407] Connection header received
0924 05:03:58.711062 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.711062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.711079 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.711123 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.711244 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.711687 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.712672 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.713014 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.713079 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.713089 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.713235 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.713271 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:03:58.713369 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.710770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46791 (local address 127.0.98.1:41541)
0924 05:03:58.710939 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.710949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.711009 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.711138 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.711139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.711141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.711185 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.711187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.711787 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.711788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.712583 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.712585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.713108 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.713109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.713173 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.713176 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.713220 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.713261 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260924 05:03:58.713369 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.710765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42917 (local address 127.0.98.1:53185)
0924 05:03:58.710892 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.710903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.710962 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.711104 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.711105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.711107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.711159 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.711160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.711756 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.711757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.712571 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.712573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.713111 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.713112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.713164 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.713167 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.713210 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.713248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":35,"thread_start_us":60,"threads_started":1}
I20260924 05:03:58.713604 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6517e3a26ac45d88d02663b42dd04b9" candidate_uuid: "b1e3c2055da640b4beca68229aa854d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07d4335ac004b61ae99f476c47ca5c7" is_pre_election: true
I20260924 05:03:58.713768 1944 raft_consensus.cc:2454] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e3c2055da640b4beca68229aa854d7 in term 0.
I20260924 05:03:58.713744 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6517e3a26ac45d88d02663b42dd04b9" candidate_uuid: "b1e3c2055da640b4beca68229aa854d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0566a286e4d5498b95e331c9dd9201d6" is_pre_election: true
I20260924 05:03:58.713821 1803 raft_consensus.cc:2454] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e3c2055da640b4beca68229aa854d7 in term 0.
I20260924 05:03:58.713963 1593 leader_election.cc:302] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e3c2055da640b4beca68229aa854d7, c07d4335ac004b61ae99f476c47ca5c7; no voters:
I20260924 05:03:58.714078 2022 raft_consensus.cc:2781] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:03:58.714111 2022 raft_consensus.cc:482] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:03:58.714138 2022 raft_consensus.cc:3037] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.714743 2022 raft_consensus.cc:504] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.714852 2022 leader_election.cc:288] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [CANDIDATE]: Term 1 election: Requested vote from peers 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917), c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791)
I20260924 05:03:58.714987 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6517e3a26ac45d88d02663b42dd04b9" candidate_uuid: "b1e3c2055da640b4beca68229aa854d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07d4335ac004b61ae99f476c47ca5c7"
I20260924 05:03:58.715026 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6517e3a26ac45d88d02663b42dd04b9" candidate_uuid: "b1e3c2055da640b4beca68229aa854d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0566a286e4d5498b95e331c9dd9201d6"
I20260924 05:03:58.715060 1944 raft_consensus.cc:3037] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.715086 1803 raft_consensus.cc:3037] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.715768 1944 raft_consensus.cc:2454] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e3c2055da640b4beca68229aa854d7 in term 1.
I20260924 05:03:58.715768 1803 raft_consensus.cc:2454] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e3c2055da640b4beca68229aa854d7 in term 1.
I20260924 05:03:58.715915 1593 leader_election.cc:302] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e3c2055da640b4beca68229aa854d7, c07d4335ac004b61ae99f476c47ca5c7; no voters:
I20260924 05:03:58.715998 2022 raft_consensus.cc:2781] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:58.716120 2022 raft_consensus.cc:686] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 1 LEADER]: Becoming Leader. State: Replica: b1e3c2055da640b4beca68229aa854d7, State: Running, Role: LEADER
I20260924 05:03:58.716209 2022 consensus_queue.cc:237] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.716813 1504 catalog_manager.cc:5495] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e3c2055da640b4beca68229aa854d7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b1e3c2055da640b4beca68229aa854d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } health_report { overall_health: UNKNOWN } } }
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:58.744453 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.738423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43109 (local address 127.0.0.1:52420)
0924 05:03:58.738590 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.738600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.738670 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.738785 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.738786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.739101 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:58.739146 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.739149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.739822 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.739823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.740341 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.740343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.740665 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.740666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.740713 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.741194 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:58.741198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:58.742486 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:58.743639 (+ 1153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:58.743644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:58.743647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:58.743839 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:58.743972 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:58.743973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:58.743973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:58.744019 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:58.744189 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:58.744193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:58.744231 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.744276 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.744358 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260924 05:03:58.744498 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.738470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.98.62:43109)
0924 05:03:58.738528 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.738530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.738667 (+ 137us) server_negotiation.cc:407] Connection header received
0924 05:03:58.738719 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.738720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.738729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.738776 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:58.739231 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.739753 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.740421 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.740608 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.740661 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.742559 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:58.742560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:58.742560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:58.742567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:58.743571 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.743889 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.743889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.743890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.743930 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.744056 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.744057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.744057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.744143 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:58.744194 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.744298 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.744398 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":31}
I20260924 05:03:58.746133 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52420:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 05:03:58.746323 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:03:58.747553 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 46d72a10b8b248ae8c0717e44709c5a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1732d4e56d3b43c0b71ab2ce7a5b9473]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:03:58.747545 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 46d72a10b8b248ae8c0717e44709c5a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1732d4e56d3b43c0b71ab2ce7a5b9473]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:03:58.747551 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 46d72a10b8b248ae8c0717e44709c5a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1732d4e56d3b43c0b71ab2ce7a5b9473]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:03:58.747679 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d72a10b8b248ae8c0717e44709c5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.747689 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d72a10b8b248ae8c0717e44709c5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.747687 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d72a10b8b248ae8c0717e44709c5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:58.748654 2018 tablet_bootstrap.cc:492] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: Bootstrap starting.
I20260924 05:03:58.748984 2018 tablet_bootstrap.cc:654] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.749051 2015 tablet_bootstrap.cc:492] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: Bootstrap starting.
I20260924 05:03:58.749408 2015 tablet_bootstrap.cc:654] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.749727 2014 tablet_bootstrap.cc:492] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: Bootstrap starting.
I20260924 05:03:58.750093 2014 tablet_bootstrap.cc:654] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:58.750128 2018 tablet_bootstrap.cc:492] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: No bootstrap required, opened a new log
I20260924 05:03:58.750164 2018 ts_tablet_manager.cc:1397] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:03:58.750274 2018 raft_consensus.cc:348] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.750326 2018 raft_consensus.cc:374] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.750341 2018 raft_consensus.cc:729] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e3c2055da640b4beca68229aa854d7, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.750387 2018 consensus_queue.cc:260] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.750468 2018 ts_tablet_manager.cc:1428] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.750485 2015 tablet_bootstrap.cc:492] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: No bootstrap required, opened a new log
I20260924 05:03:58.750526 2015 ts_tablet_manager.cc:1397] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:58.750507 2022 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.750571 2022 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.750648 2015 raft_consensus.cc:348] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.750715 2015 raft_consensus.cc:374] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.750733 2015 raft_consensus.cc:729] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07d4335ac004b61ae99f476c47ca5c7, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.750768 2015 consensus_queue.cc:260] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.750851 2015 ts_tablet_manager.cc:1428] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.750960 2024 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.751024 2024 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.751224 2014 tablet_bootstrap.cc:492] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: No bootstrap required, opened a new log
I20260924 05:03:58.751263 2014 ts_tablet_manager.cc:1397] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:03:58.751366 2014 raft_consensus.cc:348] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.751420 2014 raft_consensus.cc:374] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:58.751436 2014 raft_consensus.cc:729] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0566a286e4d5498b95e331c9dd9201d6, State: Initialized, Role: FOLLOWER
I20260924 05:03:58.751478 2014 consensus_queue.cc:260] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.751555 2014 ts_tablet_manager.cc:1428] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:58.751623 2021 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.751722 2021 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.757757 2021 raft_consensus.cc:482] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:03:58.757817 2021 raft_consensus.cc:504] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.758064 2021 leader_election.cc:288] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053), c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791)
I20260924 05:03:58.760872 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.758357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40613 (local address 127.0.98.1:33053)
0924 05:03:58.758399 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.758401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.758410 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:03:58.758482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.758483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.758493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.758549 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.758696 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.759173 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.760140 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.760586 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.760634 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.760641 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.760779 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.760810 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:03:58.760895 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.758190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33053 (local address 127.0.98.2:40613)
0924 05:03:58.758346 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.758356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.758410 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.758555 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.758555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.758557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.758636 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.758638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.759238 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.759239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.760059 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.760061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.760652 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.760653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.760718 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.760723 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.760770 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.760807 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":25,"thread_start_us":100,"threads_started":1}
I20260924 05:03:58.760946 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.758342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40327 (local address 127.0.98.3:46791)
0924 05:03:58.758397 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.758399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.758536 (+ 137us) server_negotiation.cc:407] Connection header received
0924 05:03:58.758582 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.758582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.758593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.758643 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:03:58.758761 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.759193 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.760166 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.760649 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.760711 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.760720 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.760864 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.760893 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:03:58.761056 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.758190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46791 (local address 127.0.98.2:40327)
0924 05:03:58.758466 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.758475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.758538 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.758649 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.758649 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.758651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:03:58.758715 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.758716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.759262 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.759262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.760073 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.760075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.760719 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.760720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.760796 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.760800 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.760851 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.760885 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":41,"thread_start_us":101,"threads_started":1}
I20260924 05:03:58.761113 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d72a10b8b248ae8c0717e44709c5a0" candidate_uuid: "0566a286e4d5498b95e331c9dd9201d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e3c2055da640b4beca68229aa854d7" is_pre_election: true
I20260924 05:03:58.761193 1662 raft_consensus.cc:2454] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0566a286e4d5498b95e331c9dd9201d6 in term 0.
I20260924 05:03:58.761351 1736 leader_election.cc:302] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0566a286e4d5498b95e331c9dd9201d6, b1e3c2055da640b4beca68229aa854d7; no voters:
I20260924 05:03:58.761461 2021 raft_consensus.cc:2781] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:03:58.761449 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d72a10b8b248ae8c0717e44709c5a0" candidate_uuid: "0566a286e4d5498b95e331c9dd9201d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07d4335ac004b61ae99f476c47ca5c7" is_pre_election: true
I20260924 05:03:58.761502 2021 raft_consensus.cc:482] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:03:58.761519 2021 raft_consensus.cc:3037] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.761516 1944 raft_consensus.cc:2454] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0566a286e4d5498b95e331c9dd9201d6 in term 0.
I20260924 05:03:58.762135 2021 raft_consensus.cc:504] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.762239 2021 leader_election.cc:288] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [CANDIDATE]: Term 1 election: Requested vote from peers b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053), c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791)
I20260924 05:03:58.762384 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d72a10b8b248ae8c0717e44709c5a0" candidate_uuid: "0566a286e4d5498b95e331c9dd9201d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e3c2055da640b4beca68229aa854d7"
I20260924 05:03:58.762413 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d72a10b8b248ae8c0717e44709c5a0" candidate_uuid: "0566a286e4d5498b95e331c9dd9201d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07d4335ac004b61ae99f476c47ca5c7"
I20260924 05:03:58.762455 1662 raft_consensus.cc:3037] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.762465 1944 raft_consensus.cc:3037] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:58.762966 1944 raft_consensus.cc:2454] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0566a286e4d5498b95e331c9dd9201d6 in term 1.
I20260924 05:03:58.763005 1662 raft_consensus.cc:2454] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0566a286e4d5498b95e331c9dd9201d6 in term 1.
I20260924 05:03:58.763125 1734 leader_election.cc:302] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0566a286e4d5498b95e331c9dd9201d6, c07d4335ac004b61ae99f476c47ca5c7; no voters:
I20260924 05:03:58.763216 2021 raft_consensus.cc:2781] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:58.763329 2021 raft_consensus.cc:686] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 LEADER]: Becoming Leader. State: Replica: 0566a286e4d5498b95e331c9dd9201d6, State: Running, Role: LEADER
I20260924 05:03:58.763389 2021 consensus_queue.cc:237] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } }
I20260924 05:03:58.763686 2057 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: TxnStatusTablet state changed. Reason: New leader 0566a286e4d5498b95e331c9dd9201d6. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.763816 2057 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:03:58.763983 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 05:03:58.763945 1504 catalog_manager.cc:5495] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } health_report { overall_health: UNKNOWN } } }
I20260924 05:03:58.764067 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 05:03:58.768884 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.766428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42917 (local address 127.0.0.1:56082)
0924 05:03:58.766500 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.766510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.766551 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.766717 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.766717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.766718 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:03:58.766755 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.766757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.767526 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.767527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.768070 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.768071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.768490 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.768491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.768530 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.768543 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.768779 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.768782 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.768804 (+ 22us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.768835 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 05:03:58.768890 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.766482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.0.98.2:42917)
0924 05:03:58.766521 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.766523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.766533 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:03:58.766618 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.766619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.766634 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.766706 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:58.766792 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.767455 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.768134 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.768411 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.768473 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.768612 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.768725 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.768795 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.768812 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.768829 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:03:58.769783 1662 raft_consensus.cc:1261] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Refusing update from remote peer 0566a286e4d5498b95e331c9dd9201d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:58.769929 1944 raft_consensus.cc:1261] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Refusing update from remote peer 0566a286e4d5498b95e331c9dd9201d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:58.770064 2021 consensus_queue.cc:1035] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:58.770138 2057 consensus_queue.cc:1035] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:58.770669 2022 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: TxnStatusTablet state changed. Reason: New leader 0566a286e4d5498b95e331c9dd9201d6. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.770745 2022 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.770746 2024 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: TxnStatusTablet state changed. Reason: New leader 0566a286e4d5498b95e331c9dd9201d6. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.770816 2024 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.771127 2021 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.771207 2021 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:03:58.771898 2022 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.771975 2022 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P b1e3c2055da640b4beca68229aa854d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.775034 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.772616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.0.98.1:33053)
0924 05:03:58.772647 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.772648 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.772658 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:03:58.772735 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.772736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.772748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.772798 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:58.772897 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.773316 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.774335 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.774529 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.774581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.774727 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.774850 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.774899 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.774961 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.774976 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:03:58.780416 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.777673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43749 (local address 127.0.98.2:42917)
0924 05:03:58.777736 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.777738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.777837 (+ 99us) server_negotiation.cc:407] Connection header received
0924 05:03:58.777859 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.777860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.777868 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.777926 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:58.778020 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.778709 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.779747 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.780035 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.780096 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.780224 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.780252 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.780292 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.780336 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.780350 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:03:58.780421 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.777611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42917 (local address 127.0.98.1:43749)
0924 05:03:58.777767 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.777776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.777828 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.777929 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.777929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.777930 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:03:58.777981 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.777982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.778781 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.778781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.779674 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.779676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.780101 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.780102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.780146 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.780157 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.780301 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.780302 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.780329 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.780357 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260924 05:03:58.781579 2057 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.781670 2057 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P 0566a286e4d5498b95e331c9dd9201d6: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:03:58.782006 2024 tablet_replica.cc:439] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0566a286e4d5498b95e331c9dd9201d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e3c2055da640b4beca68229aa854d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33053 } } peers { permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 } } peers { permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 } } }
I20260924 05:03:58.782081 2024 tablet_replica.cc:442] T 46d72a10b8b248ae8c0717e44709c5a0 P c07d4335ac004b61ae99f476c47ca5c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:03:58.782917 1944 raft_consensus.cc:1261] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Refusing update from remote peer b1e3c2055da640b4beca68229aa854d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:58.783005 1803 raft_consensus.cc:1261] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Refusing update from remote peer b1e3c2055da640b4beca68229aa854d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:58.783120 2022 consensus_queue.cc:1035] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07d4335ac004b61ae99f476c47ca5c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:58.783258 2022 consensus_queue.cc:1035] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0566a286e4d5498b95e331c9dd9201d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:58.791085 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.788555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46789 (local address 127.0.98.1:33053)
0924 05:03:58.788589 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.788591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.788760 (+ 169us) server_negotiation.cc:407] Connection header received
0924 05:03:58.788814 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.788815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.788824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.788872 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:58.788998 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.789438 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.790335 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.790563 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.790594 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.790870 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.790901 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.790939 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.791004 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.791020 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:03:58.791131 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.788524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33053 (local address 127.0.98.2:46789)
0924 05:03:58.788711 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:58.788721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:58.788771 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:58.788875 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:58.788876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:58.788877 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:03:58.788936 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.788937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.789513 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.789514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.790243 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:58.790246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.790691 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.790691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:58.790756 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.790772 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:03:58.790960 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:03:58.790962 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:03:58.791002 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:03:58.791042 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260924 05:03:58.800575 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:03:58.800678 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:03:58.801328 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6170ce1d07c482ea768f907147a4225: Sending DeleteTablet for 3 replicas of tablet a6517e3a26ac45d88d02663b42dd04b9
I20260924 05:03:58.801620 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet a6517e3a26ac45d88d02663b42dd04b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41586
I20260924 05:03:58.801620 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a6517e3a26ac45d88d02663b42dd04b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44210
I20260924 05:03:58.801730 2036 tablet_replica.cc:330] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: stopping tablet replica
I20260924 05:03:58.801736 2037 tablet_replica.cc:330] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: stopping tablet replica
I20260924 05:03:58.801770 2036 raft_consensus.cc:2229] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:03:58.801782 2037 raft_consensus.cc:2229] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:03:58.801803 2036 raft_consensus.cc:2258] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:03:58.801853 2037 raft_consensus.cc:2258] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:03:58.802266 2036 ts_tablet_manager.cc:1905] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:03:58.802901 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet a6517e3a26ac45d88d02663b42dd04b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:03:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56066
I20260924 05:03:58.803177 2038 tablet_replica.cc:330] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: stopping tablet replica
I20260924 05:03:58.803249 2038 raft_consensus.cc:2229] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:03:58.803284 2038 raft_consensus.cc:2258] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:03:58.803611 2038 ts_tablet_manager.cc:1905] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:03:58.803807 2036 ts_tablet_manager.cc:1918] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:03:58.803853 2036 log.cc:1196] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/a6517e3a26ac45d88d02663b42dd04b9
I20260924 05:03:58.804041 2036 ts_tablet_manager.cc:1939] T a6517e3a26ac45d88d02663b42dd04b9 P c07d4335ac004b61ae99f476c47ca5c7: Deleting consensus metadata
I20260924 05:03:58.804605 2037 ts_tablet_manager.cc:1905] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:03:58.805085 2038 ts_tablet_manager.cc:1918] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:03:58.805130 2038 log.cc:1196] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/a6517e3a26ac45d88d02663b42dd04b9
I20260924 05:03:58.805310 2038 ts_tablet_manager.cc:1939] T a6517e3a26ac45d88d02663b42dd04b9 P 0566a286e4d5498b95e331c9dd9201d6: Deleting consensus metadata
I20260924 05:03:58.805517 1489 catalog_manager.cc:4841] TS c07d4335ac004b61ae99f476c47ca5c7 (127.0.98.3:46791): tablet a6517e3a26ac45d88d02663b42dd04b9 (table test-table [id=e503da520d0940a5b116019362311b3b]) successfully deleted
I20260924 05:03:58.806131 2037 ts_tablet_manager.cc:1918] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:03:58.806176 2037 log.cc:1196] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/a6517e3a26ac45d88d02663b42dd04b9
I20260924 05:03:58.806354 2037 ts_tablet_manager.cc:1939] T a6517e3a26ac45d88d02663b42dd04b9 P b1e3c2055da640b4beca68229aa854d7: Deleting consensus metadata
I20260924 05:03:58.806639 1491 catalog_manager.cc:4841] TS b1e3c2055da640b4beca68229aa854d7 (127.0.98.1:33053): tablet a6517e3a26ac45d88d02663b42dd04b9 (table test-table [id=e503da520d0940a5b116019362311b3b]) successfully deleted
I20260924 05:03:58.806854 1490 catalog_manager.cc:4841] TS 0566a286e4d5498b95e331c9dd9201d6 (127.0.98.2:42917): tablet a6517e3a26ac45d88d02663b42dd04b9 (table test-table [id=e503da520d0940a5b116019362311b3b]) successfully deleted
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:03:58.819183 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.803214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.0.98.1:33053)
0924 05:03:58.803246 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.803247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.809888 (+ 6641us) server_negotiation.cc:407] Connection header received
0924 05:03:58.809937 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.809938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.809948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.810006 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:58.810638 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.811368 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.812408 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.812711 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.812771 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.815998 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:58.816005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:58.816005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:58.816021 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:58.817895 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.818263 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.818266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.818266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.818344 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.818528 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.818529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.818529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.818686 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:58.818829 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.818997 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.819077 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:03:58.819567 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41616
I20260924 05:03:58.827695 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:58.820531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.62:43109)
0924 05:03:58.820586 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:58.820587 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:58.820696 (+ 109us) server_negotiation.cc:407] Connection header received
0924 05:03:58.820744 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:58.820745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:58.820756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:58.820814 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:58.821408 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.822237 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.823263 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:58.823567 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:58.823632 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:58.824961 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:58.824963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:58.824963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:58.824971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:58.826539 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.826902 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.826903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.826904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.826964 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:58.827174 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:58.827174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:58.827175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:58.827307 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:58.827403 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:58.827509 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 05:03:58.827587 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:03:58.827993 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52424
I20260924 05:03:58.828718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1576
I20260924 05:03:58.834431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1717
I20260924 05:03:58.839268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1858
I20260924 05:03:58.843600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 1472
2026-09-24T05:03:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6700 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[2081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[2081](info): set up 2 sockets
Sep 24 05:03:58 dist-test-slave-m01t krb5kdc[2081](info): commencing operation
krb5kdc: starting...
W20260924 05:03:59.777024 392 mini_kdc.cc:121] Time spent starting KDC: real 0.915s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:59 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:59.880627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:59.957469 2094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:59.957653 2094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:59.957674 2094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:59.958781 2094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:59.958820 2094 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 05:03:59.958832 2094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:59.958844 2094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:59.958858 2094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:59.958868 2094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:59.960228 2094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36217
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:59.960403 2094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:59.960578 2094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:59.964926 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:59.965025 2094 server_base.cc:1034] running on GCE node
W20260924 05:03:59.964958 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:59.964949 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:59.965348 2094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:59.965606 2094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:59.966733 2094 hybrid_clock.cc:648] HybridClock initialized: now 1790226239966719 us; error 29 us; skew 500 ppm
Sep 24 05:03:59 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:59.968876 2094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:59.969251 2094 webserver.cc:460] Webserver started at http://127.0.98.62:41731/ using document root <none> and password file <none>
I20260924 05:03:59.969449 2094 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:59.969494 2094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:59.969615 2094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:59.970624 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "007d5f161bdf48e091026c45fd241ca9"
format_stamp: "Formatted at 2026-09-24 05:03:59 on dist-test-slave-m01t"
server_key: "200ab87e97c1f0bb899ef3b9c2002c58"
server_key_iv: "21fa6d47a329ae971a3dba3d901c7ad3"
server_key_version: "encryptionkey@0"
I20260924 05:03:59.970947 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "007d5f161bdf48e091026c45fd241ca9"
format_stamp: "Formatted at 2026-09-24 05:03:59 on dist-test-slave-m01t"
server_key: "200ab87e97c1f0bb899ef3b9c2002c58"
server_key_iv: "21fa6d47a329ae971a3dba3d901c7ad3"
server_key_version: "encryptionkey@0"
I20260924 05:03:59.972079 2094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:59.972663 2109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:59.972805 2094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:59.972873 2094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "007d5f161bdf48e091026c45fd241ca9"
format_stamp: "Formatted at 2026-09-24 05:03:59 on dist-test-slave-m01t"
server_key: "200ab87e97c1f0bb899ef3b9c2002c58"
server_key_iv: "21fa6d47a329ae971a3dba3d901c7ad3"
server_key_version: "encryptionkey@0"
I20260924 05:03:59.972935 2094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:59.985903 2094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:59.987449 2094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:59.987571 2094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:59.991667 2094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36217
I20260924 05:03:59.991719 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36217 every 8 connection(s)
I20260924 05:03:59.992024 2094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:59.992642 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:59.994030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 2094
I20260924 05:03:59.994103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:59.994243 392 external_mini_cluster.cc:1428] Setting key 0a209254bdebda91a3b4d993e82a0672
I20260924 05:03:59.995172 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Bootstrap starting.
I20260924 05:03:59.995846 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:59.996114 2172 log.cc:824] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:59.996778 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: No bootstrap required, opened a new log
I20260924 05:03:59.999011 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d5f161bdf48e091026c45fd241ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36217 } }
I20260924 05:03:59.999197 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:59.999235 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007d5f161bdf48e091026c45fd241ca9, State: Initialized, Role: FOLLOWER
I20260924 05:03:59.999360 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d5f161bdf48e091026c45fd241ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36217 } }
I20260924 05:03:59.999431 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:59.999462 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:59.999508 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:59.999995 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d5f161bdf48e091026c45fd241ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36217 } }
I20260924 05:04:00.000094 2172 leader_election.cc:302] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 007d5f161bdf48e091026c45fd241ca9; no voters:
I20260924 05:04:00.000243 2172 leader_election.cc:288] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:00.000272 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:00.000375 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 007d5f161bdf48e091026c45fd241ca9, State: Running, Role: LEADER
I20260924 05:04:00.000504 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:00.000526 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d5f161bdf48e091026c45fd241ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36217 } }
Sep 24 05:03:59 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:00.000905 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 007d5f161bdf48e091026c45fd241ca9. Latest consensus state: current_term: 1 leader_uuid: "007d5f161bdf48e091026c45fd241ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d5f161bdf48e091026c45fd241ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36217 } } }
I20260924 05:04:00.000983 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:00.001348 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "007d5f161bdf48e091026c45fd241ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d5f161bdf48e091026c45fd241ca9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36217 } } }
I20260924 05:04:00.001430 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:00.001904 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:00.002396 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:00.004047 2186 catalog_manager.cc:1347] Generated new cluster ID: b27e7192e4a841dc8d0d7ef4609dcf5a
I20260924 05:04:00.004098 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:00.004930 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:59.994716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.0.98.62:36217)
0924 05:03:59.994896 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:59.994907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:59.994938 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:03:59.995438 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:59.995449 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:59.995518 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:59.995681 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:59.996232 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:59.997531 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:59.998294 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:59.998910 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:59.999023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.001328 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.001332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.001332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.001345 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.003473 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.003781 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.003784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.003785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.003869 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.004054 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.004055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.004055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.004354 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.004482 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.004623 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.004747 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260924 05:04:00.014938 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:00.015427 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:00.022428 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Generated new TSK 0
I20260924 05:04:00.022625 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:00.054015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36217
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:00.149430 2198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:00.149627 2198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:00.149649 2198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:00.150776 2198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:00.150817 2198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:00.150838 2198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 05:04:00.150861 2198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:04:00.150887 2198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:00.152261 2198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:00.152467 2198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:00.152676 2198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:00.156985 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:00.156980 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:00.156977 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:00.157287 2198 server_base.cc:1034] running on GCE node
I20260924 05:04:00.157485 2198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:00.157737 2198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:00.158878 2198 hybrid_clock.cc:648] HybridClock initialized: now 1790226240158866 us; error 28 us; skew 500 ppm
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:00.161087 2198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:00.161495 2198 webserver.cc:460] Webserver started at http://127.0.98.1:46553/ using document root <none> and password file <none>
I20260924 05:04:00.161700 2198 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:00.161780 2198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:00.161906 2198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:00.162931 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "77c8531a5c154284b0481b311c7441db"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "7e3655e933367a4797bd249256deaca8"
server_key_iv: "6e1d331f5651feb03a9cf3f046f426c4"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.163264 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77c8531a5c154284b0481b311c7441db"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "7e3655e933367a4797bd249256deaca8"
server_key_iv: "6e1d331f5651feb03a9cf3f046f426c4"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.164438 2198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:00.165043 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.165190 2198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:00.165268 2198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "77c8531a5c154284b0481b311c7441db"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "7e3655e933367a4797bd249256deaca8"
server_key_iv: "6e1d331f5651feb03a9cf3f046f426c4"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.165333 2198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:00.189060 2198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:00.190887 2198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:00.191031 2198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:00.191546 2198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:00.191596 2198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.191627 2198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:00.191643 2198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.197844 2198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44247
I20260924 05:04:00.197918 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44247 every 8 connection(s)
I20260924 05:04:00.198213 2198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:00.198841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 2198
I20260924 05:04:00.198936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:00.199113 392 external_mini_cluster.cc:1428] Setting key 541c7fc3191c506dbd970eb87cf48682
I20260924 05:04:00.201498 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.191953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51047 (local address 127.0.98.62:36217)
0924 05:04:00.191994 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.191996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.192227 (+ 231us) server_negotiation.cc:407] Connection header received
0924 05:04:00.192715 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.192716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.192728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.192783 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.193569 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.194366 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.195886 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.196186 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.196238 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.198745 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.198747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.198747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.198754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.200123 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.200544 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.200545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.200545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.200594 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.200829 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.200830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.200830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.200960 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.201023 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.201302 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.201398 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:00.201567 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.191958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.98.1:51047)
0924 05:04:00.192103 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.192137 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.192624 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.192860 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.192864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.193195 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.193474 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.193482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.194481 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.194482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.195778 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.195782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.196266 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.196267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.196413 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.196959 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.196969 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.198664 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.200192 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.200198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.200200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.200480 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.200643 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.200643 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.200644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.200768 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.201016 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.201020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.201152 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.201247 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.201392 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260924 05:04:00.202047 2329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36217
I20260924 05:04:00.202150 2329 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:00.202395 2329 heartbeater.cc:511] Master 127.0.98.62:36217 requested a full tablet report, sending...
I20260924 05:04:00.202992 2125 ts_manager.cc:194] Registered new tserver with Master: 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247)
I20260924 05:04:00.203579 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51047
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:04:00.208288 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.203124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56509 (local address 127.0.98.62:36217)
0924 05:04:00.203156 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.203158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.203264 (+ 106us) server_negotiation.cc:407] Connection header received
0924 05:04:00.203356 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.203356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.203365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.203398 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.203773 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.204267 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.205289 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.205463 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.205513 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.206372 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.206372 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.206373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.206377 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.207492 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.207770 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.207770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.207771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.207811 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.207956 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.207957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.207957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.208036 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.208076 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.208181 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.208233 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:00.208304 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.202964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.98.1:56509)
0924 05:04:00.203207 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.203217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.203295 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.203457 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.203458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.203653 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.203709 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.203711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.204357 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.204358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.205210 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.205213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.205530 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.205531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.205614 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.205968 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.205973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.206297 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.207556 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.207557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.207558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.207704 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.207853 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.207854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.207855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.207916 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.208091 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.208091 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.208122 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.208166 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.208236 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":29,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:00.221838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36217
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:00.300230 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:00.300498 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:00.300522 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:00.301744 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:00.301791 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:00.301815 2338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 05:04:00.301836 2338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:04:00.301863 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:00.303351 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:00.303561 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:00.303757 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:00.309021 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:00.309059 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:00.309088 2338 server_base.cc:1034] running on GCE node
W20260924 05:04:00.309026 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:00.309420 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:00.309643 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:00.310779 2338 hybrid_clock.cc:648] HybridClock initialized: now 1790226240310778 us; error 40 us; skew 500 ppm
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:00.313004 2338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:00.313410 2338 webserver.cc:460] Webserver started at http://127.0.98.2:41955/ using document root <none> and password file <none>
I20260924 05:04:00.313603 2338 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:00.313648 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:00.313787 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:00.314769 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "65115da28609498692bd7ef4c3771995"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "47e7af2dd4a6cc62bf6b5c989013c41c"
server_key_iv: "10696d0e747bff2713fb65c848d2b4bc"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.315088 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "65115da28609498692bd7ef4c3771995"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "47e7af2dd4a6cc62bf6b5c989013c41c"
server_key_iv: "10696d0e747bff2713fb65c848d2b4bc"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.316246 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:00.316820 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.316946 2338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:00.317004 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "65115da28609498692bd7ef4c3771995"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "47e7af2dd4a6cc62bf6b5c989013c41c"
server_key_iv: "10696d0e747bff2713fb65c848d2b4bc"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.317066 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:00.340183 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:00.341938 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:00.342067 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:00.342561 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:00.342608 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.342629 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:00.342641 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.348749 2338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45111
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:00.348809 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45111 every 8 connection(s)
I20260924 05:04:00.349130 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:00.351468 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.343017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54175 (local address 127.0.98.62:36217)
0924 05:04:00.343082 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.343084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.343267 (+ 183us) server_negotiation.cc:407] Connection header received
0924 05:04:00.343736 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.343738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.343751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.343822 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.344606 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.345350 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.346502 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.346801 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.346877 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.349278 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.349279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.349280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.349288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.350532 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.350844 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.350845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.350845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.350886 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.351037 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.351038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.351038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.351123 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.351166 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.351357 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.351407 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:04:00.351644 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.342968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.98.2:54175)
0924 05:04:00.343145 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.343184 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.343649 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.343887 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.343891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.344235 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.344506 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.344513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.345441 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.345442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.346432 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.346435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.346892 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.346892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.347034 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.347552 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.347563 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.349184 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.350594 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.350597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.350598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.350789 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.350920 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.350921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.350921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.351000 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.351164 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.351166 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.351270 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.351338 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.351444 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260924 05:04:00.352020 2469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36217
I20260924 05:04:00.352105 2469 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:00.352275 2469 heartbeater.cc:511] Master 127.0.98.62:36217 requested a full tablet report, sending...
I20260924 05:04:00.352599 2126 ts_manager.cc:194] Registered new tserver with Master: 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111)
I20260924 05:04:00.353037 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54175
I20260924 05:04:00.356081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 2338
I20260924 05:04:00.356169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:00.356328 392 external_mini_cluster.cc:1428] Setting key 6dcd8507fe8ce648954176b2ba39ee36
I20260924 05:04:00.357787 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.352932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.98.2:34089)
0924 05:04:00.353051 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.353061 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.353124 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.353237 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.353237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.353372 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.353418 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.353419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.354066 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.354067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.354547 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.354548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.354850 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.354850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.354916 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.355269 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.355273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.355508 (+ 235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.356948 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.356948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.356948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.357098 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.357267 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.357268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.357268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.357307 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.357537 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.357537 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.357576 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.357618 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.357680 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260924 05:04:00.357801 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.352984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34089 (local address 127.0.98.62:36217)
0924 05:04:00.353015 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.353017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.353112 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:04:00.353167 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.353168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.353176 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.353231 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.353484 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.353989 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.354611 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.354796 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.354845 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.355580 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.355581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.355581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.355587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.356883 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.357161 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.357162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.357163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.357214 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.357338 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.357338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.357339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.357487 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.357544 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.357640 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.357705 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:00.376412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36217
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:00.454594 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:00.454789 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:00.454811 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:00.455892 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:00.455931 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:00.455951 2478 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 05:04:00.455973 2478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:04:00.456003 2478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:00.457388 2478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:00.457594 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:00.457809 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:00.462252 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:00.462244 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:00.462244 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:00.462503 2478 server_base.cc:1034] running on GCE node
I20260924 05:04:00.462677 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:00.462910 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:00.464051 2478 hybrid_clock.cc:648] HybridClock initialized: now 1790226240464039 us; error 28 us; skew 500 ppm
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:00.466266 2478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:00.466645 2478 webserver.cc:460] Webserver started at http://127.0.98.3:33987/ using document root <none> and password file <none>
I20260924 05:04:00.466826 2478 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:00.466871 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:00.466987 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:00.468042 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "f76e84dca8c3454288c93c1f03c306c5"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "c1799ae7a7146a5388212ff08a4e47aa"
server_key_iv: "516db52c43a670d671d5b74127f30be1"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.468374 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f76e84dca8c3454288c93c1f03c306c5"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "c1799ae7a7146a5388212ff08a4e47aa"
server_key_iv: "516db52c43a670d671d5b74127f30be1"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.469482 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:00.470121 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.470280 2478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.470355 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "f76e84dca8c3454288c93c1f03c306c5"
format_stamp: "Formatted at 2026-09-24 05:04:00 on dist-test-slave-m01t"
server_key: "c1799ae7a7146a5388212ff08a4e47aa"
server_key_iv: "516db52c43a670d671d5b74127f30be1"
server_key_version: "encryptionkey@0"
I20260924 05:04:00.470422 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:00.480899 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:00.482683 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:00.482802 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:00.483214 2478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:00.483249 2478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.483275 2478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:00.483301 2478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.489066 2478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46855
I20260924 05:04:00.489132 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46855 every 8 connection(s)
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:00.489440 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:00.490342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 2478
I20260924 05:04:00.490445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:00.490661 392 external_mini_cluster.cc:1428] Setting key eb53b0cd8d3e4079a20b05daa0646d80
I20260924 05:04:00.492344 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.483604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39841 (local address 127.0.98.62:36217)
0924 05:04:00.483677 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.483679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.483897 (+ 218us) server_negotiation.cc:407] Connection header received
0924 05:04:00.484334 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.484335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.484348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.484423 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.485133 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.485890 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.487285 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.487557 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.487635 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.489963 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.489965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.489965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.489973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.491284 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.491685 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.491686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.491686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.491737 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.491898 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.491898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.491899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.491991 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.492031 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.492228 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.492282 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 05:04:00.492518 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.483567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.98.3:39841)
0924 05:04:00.483777 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.483815 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.484254 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.484490 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.484493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.484788 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.485044 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.485049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.485980 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.485981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.487201 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.487203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.487634 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.487635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.487724 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.488216 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.488226 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.489878 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.491357 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.491364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.491366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.491622 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.491776 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.491776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.491776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.491859 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.492028 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.492030 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.492138 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.492208 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.492321 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":136,"threads_started":1}
I20260924 05:04:00.492931 2609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36217
I20260924 05:04:00.493010 2609 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:00.493197 2609 heartbeater.cc:511] Master 127.0.98.62:36217 requested a full tablet report, sending...
I20260924 05:04:00.493531 2125 ts_manager.cc:194] Registered new tserver with Master: f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:00.493953 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39841
I20260924 05:04:00.498502 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.493967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54795 (local address 127.0.98.62:36217)
0924 05:04:00.494007 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.494008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.494089 (+ 81us) server_negotiation.cc:407] Connection header received
0924 05:04:00.494167 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.494168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.494177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.494228 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.494503 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.494963 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.495604 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.495808 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.495859 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.496585 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.496586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.496586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.496592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.497525 (+ 933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.497861 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.497862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.497862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.497911 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.498059 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.498060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.498060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.498166 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.498219 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.498346 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.498410 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:00.498514 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.493907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.98.3:54795)
0924 05:04:00.494032 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.494042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.494121 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.494233 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.494234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.494391 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.494439 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.494440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.495044 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.495045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.495537 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.495538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.495864 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.495865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.495938 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.496265 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.496268 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.496511 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.497597 (+ 1086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.497598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.497599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.497797 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.497958 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.497959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.497959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.498011 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.498218 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.498218 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.498259 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.498319 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.498411 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260924 05:04:00.502146 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:00.521307 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.515694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.62:36217)
0924 05:04:00.515750 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.515752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.515761 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:00.515782 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.515782 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.515791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.515835 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.516308 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.516774 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.517415 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.517579 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.517618 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.519454 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.519455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.519455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.519463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.520484 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.520790 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.520791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.520792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.520836 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.520960 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.520960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.520961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.521049 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.521117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.521185 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.521238 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:04:00.522699 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:00.524221 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:00.530450 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.526431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46855 (local address 127.0.0.1:35516)
0924 05:04:00.526736 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.526745 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.526834 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.527188 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.527188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.527190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.527215 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.527216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.528041 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.528042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.529247 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.529250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.530214 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.530214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.530272 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.530276 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.530323 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.530365 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260924 05:04:00.530475 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.526535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.0.98.3:46855)
0924 05:04:00.526669 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.526674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.526806 (+ 132us) server_negotiation.cc:407] Connection header received
0924 05:04:00.526918 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.526920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.526933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.527059 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.527296 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.527965 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.529352 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.530053 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.530120 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.530181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.530348 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.530387 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 05:04:00.530542 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.526593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.98.2:45111)
0924 05:04:00.526752 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.526756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.526765 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:00.526802 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.526804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.526819 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.526895 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.527177 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.528269 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.529480 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.530001 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.530066 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.530127 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.530410 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.530452 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260924 05:04:00.530646 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.526401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.0.98.1:44247)
0924 05:04:00.526645 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.526650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.526754 (+ 104us) server_negotiation.cc:407] Connection header received
0924 05:04:00.526814 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.526816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.526832 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.526902 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.527132 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.528269 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.529302 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.530054 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.530120 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.530174 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.530496 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.530541 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":73,"threads_started":1}
I20260924 05:04:00.530371 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.526256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45111 (local address 127.0.0.1:41528)
0924 05:04:00.526414 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.526455 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.526516 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.527087 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.527088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.527089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.527125 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.527125 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.528349 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.528350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.529414 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.529416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.530079 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.530080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.530156 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.530169 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.530234 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.530288 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":26,"thread_start_us":53,"threads_started":1}
I20260924 05:04:00.530776 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.526268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.0.1:41954)
0924 05:04:00.526681 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.526691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.526752 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.526929 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.526932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.526934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.526997 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.526999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.528363 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.528364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.529232 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.529234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.530318 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.530319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.530377 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.530381 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.530428 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.530459 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":70,"threads_started":1}
I20260924 05:04:00.531607 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 88e5967d3ae3451e8fc66384cb86cca5 (DEFAULT_TABLE table=test-table [id=f067b38ebe51418d82ba7bbe07c7da52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:00.531606 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 88e5967d3ae3451e8fc66384cb86cca5 (DEFAULT_TABLE table=test-table [id=f067b38ebe51418d82ba7bbe07c7da52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:00.531966 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e5967d3ae3451e8fc66384cb86cca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.531966 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e5967d3ae3451e8fc66384cb86cca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.535297 2632 tablet_bootstrap.cc:492] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Bootstrap starting.
I20260924 05:04:00.535688 2633 tablet_bootstrap.cc:492] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Bootstrap starting.
I20260924 05:04:00.535996 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 88e5967d3ae3451e8fc66384cb86cca5 (DEFAULT_TABLE table=test-table [id=f067b38ebe51418d82ba7bbe07c7da52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:00.536319 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e5967d3ae3451e8fc66384cb86cca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.536345 2632 tablet_bootstrap.cc:654] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.536345 2633 tablet_bootstrap.cc:654] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.536664 2632 log.cc:824] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:00.536664 2633 log.cc:824] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:00.537400 2632 tablet_bootstrap.cc:492] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: No bootstrap required, opened a new log
I20260924 05:04:00.537400 2633 tablet_bootstrap.cc:492] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: No bootstrap required, opened a new log
I20260924 05:04:00.537470 2633 ts_tablet_manager.cc:1397] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:00.537469 2632 ts_tablet_manager.cc:1397] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:00.539222 2636 tablet_bootstrap.cc:492] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Bootstrap starting.
I20260924 05:04:00.539390 2633 raft_consensus.cc:348] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.539390 2632 raft_consensus.cc:348] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.539563 2633 raft_consensus.cc:374] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.539573 2632 raft_consensus.cc:374] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.539598 2633 raft_consensus.cc:729] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.539606 2632 raft_consensus.cc:729] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76e84dca8c3454288c93c1f03c306c5, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.539738 2632 consensus_queue.cc:260] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.539739 2633 consensus_queue.cc:260] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.539852 2636 tablet_bootstrap.cc:654] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.539999 2633 ts_tablet_manager.cc:1428] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:00.539999 2632 ts_tablet_manager.cc:1428] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:04:00.540125 2329 heartbeater.cc:503] Master 127.0.98.62:36217 was elected leader, sending a full tablet report...
I20260924 05:04:00.540130 2609 heartbeater.cc:503] Master 127.0.98.62:36217 was elected leader, sending a full tablet report...
I20260924 05:04:00.540162 2636 log.cc:824] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:00.540916 2636 tablet_bootstrap.cc:492] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: No bootstrap required, opened a new log
I20260924 05:04:00.540975 2636 ts_tablet_manager.cc:1397] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:00.542510 2636 raft_consensus.cc:348] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.542634 2636 raft_consensus.cc:374] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.542663 2636 raft_consensus.cc:729] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65115da28609498692bd7ef4c3771995, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.542765 2636 consensus_queue.cc:260] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.542977 2636 ts_tablet_manager.cc:1428] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:00.543027 2469 heartbeater.cc:503] Master 127.0.98.62:36217 was elected leader, sending a full tablet report...
W20260924 05:04:00.599666 2470 tablet.cc:2369] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:04:00.698902 2330 tablet.cc:2369] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:00.699347 2638 raft_consensus.cc:482] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:00.699445 2638 raft_consensus.cc:504] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.699734 2638 leader_election.cc:288] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247)
I20260924 05:04:00.703261 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.699969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48063 (local address 127.0.98.1:44247)
0924 05:04:00.700052 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.700054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.700064 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:00.700093 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.700094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.700105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.700182 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.700339 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.700897 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.702340 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.702918 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.702985 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.702995 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.703172 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.703205 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:04:00.703260 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.700024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40481 (local address 127.0.98.2:45111)
0924 05:04:00.700110 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.700112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.700273 (+ 161us) server_negotiation.cc:407] Connection header received
0924 05:04:00.700301 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.700302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.700313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.700362 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.700498 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.700971 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.702341 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.702915 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.702985 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.702995 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.703150 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.703192 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 05:04:00.703276 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.699918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.98.3:48063)
0924 05:04:00.699962 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.699973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.700013 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.700189 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.700190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.700192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.700259 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.700261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.700961 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.700962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.702270 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.702272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.702986 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.702987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.703080 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.703084 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.703140 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.703182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":44}
I20260924 05:04:00.703285 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.699918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45111 (local address 127.0.98.3:40481)
0924 05:04:00.700208 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.700217 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.700267 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.700369 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.700370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.700372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.700433 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.700434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.701040 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.701041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.702269 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.702272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.702986 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.702987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.703080 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.703084 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.703140 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.703182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":137,"thread_start_us":105,"threads_started":1}
I20260924 05:04:00.703634 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e5967d3ae3451e8fc66384cb86cca5" candidate_uuid: "f76e84dca8c3454288c93c1f03c306c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c8531a5c154284b0481b311c7441db" is_pre_election: true
I20260924 05:04:00.703747 2283 raft_consensus.cc:2454] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76e84dca8c3454288c93c1f03c306c5 in term 0.
I20260924 05:04:00.703738 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e5967d3ae3451e8fc66384cb86cca5" candidate_uuid: "f76e84dca8c3454288c93c1f03c306c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995" is_pre_election: true
I20260924 05:04:00.703861 2423 raft_consensus.cc:2454] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76e84dca8c3454288c93c1f03c306c5 in term 0.
I20260924 05:04:00.703971 2494 leader_election.cc:302] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c8531a5c154284b0481b311c7441db, f76e84dca8c3454288c93c1f03c306c5; no voters:
I20260924 05:04:00.704092 2638 raft_consensus.cc:2781] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:00.704147 2638 raft_consensus.cc:482] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:00.704176 2638 raft_consensus.cc:3037] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.704823 2638 raft_consensus.cc:504] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.704943 2638 leader_election.cc:288] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [CANDIDATE]: Term 1 election: Requested vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247)
I20260924 05:04:00.705076 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e5967d3ae3451e8fc66384cb86cca5" candidate_uuid: "f76e84dca8c3454288c93c1f03c306c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c8531a5c154284b0481b311c7441db"
I20260924 05:04:00.705144 2283 raft_consensus.cc:3037] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.705770 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e5967d3ae3451e8fc66384cb86cca5" candidate_uuid: "f76e84dca8c3454288c93c1f03c306c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995"
I20260924 05:04:00.705789 2283 raft_consensus.cc:2454] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76e84dca8c3454288c93c1f03c306c5 in term 1.
I20260924 05:04:00.705837 2423 raft_consensus.cc:3037] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.705938 2494 leader_election.cc:302] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c8531a5c154284b0481b311c7441db, f76e84dca8c3454288c93c1f03c306c5; no voters:
I20260924 05:04:00.706017 2638 raft_consensus.cc:2781] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:00.706161 2638 raft_consensus.cc:686] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 1 LEADER]: Becoming Leader. State: Replica: f76e84dca8c3454288c93c1f03c306c5, State: Running, Role: LEADER
I20260924 05:04:00.706261 2638 consensus_queue.cc:237] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.706650 2423 raft_consensus.cc:2454] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76e84dca8c3454288c93c1f03c306c5 in term 1.
I20260924 05:04:00.707042 2126 catalog_manager.cc:5495] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f76e84dca8c3454288c93c1f03c306c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } health_report { overall_health: HEALTHY } } }
Sep 24 05:04:00 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:00.737877 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.726423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.0.98.62:36217)
0924 05:04:00.726502 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.726505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.726575 (+ 70us) server_negotiation.cc:407] Connection header received
0924 05:04:00.726619 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.726619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.726628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.726677 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:00.727088 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.727561 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.728194 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.728364 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.728405 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.735709 (+ 7304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:00.735711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:00.735711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:00.735718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:00.736786 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.737132 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.737133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.737134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.737181 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:00.737329 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:00.737330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:00.737330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:00.737434 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:00.737487 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.737639 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.737751 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 05:04:00.737934 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.726364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36217 (local address 127.0.0.1:44794)
0924 05:04:00.726501 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.726510 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.726578 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.726688 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.726688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.726982 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:00.727027 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.727028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.727619 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.727620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.728137 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.728139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.728410 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.728411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.728446 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.734200 (+ 5754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:00.734205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:00.735601 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:00.736863 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.736873 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.736876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.737066 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.737232 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:00.737232 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:00.737233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:00.737282 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:00.737485 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:00.737494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:00.737541 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.737610 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.737809 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":89,"thread_start_us":68,"threads_started":1}
I20260924 05:04:00.739786 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44794:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 05:04:00.740000 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 05:04:00.740183 2610 tablet.cc:2369] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:00.741294 2263 tablet_service.cc:1467] Processing CreateTablet for tablet decc8d3488de45d7bd7075085e7a66a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b1a013a723a49d0a228a06712f769b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:04:00.741298 2543 tablet_service.cc:1467] Processing CreateTablet for tablet decc8d3488de45d7bd7075085e7a66a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b1a013a723a49d0a228a06712f769b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:04:00.741431 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decc8d3488de45d7bd7075085e7a66a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.741434 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decc8d3488de45d7bd7075085e7a66a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.741870 2403 tablet_service.cc:1467] Processing CreateTablet for tablet decc8d3488de45d7bd7075085e7a66a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b1a013a723a49d0a228a06712f769b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:04:00.742017 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decc8d3488de45d7bd7075085e7a66a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.742653 2632 tablet_bootstrap.cc:492] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: Bootstrap starting.
I20260924 05:04:00.742810 2633 tablet_bootstrap.cc:492] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: Bootstrap starting.
I20260924 05:04:00.743072 2632 tablet_bootstrap.cc:654] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.743155 2633 tablet_bootstrap.cc:654] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.743441 2636 tablet_bootstrap.cc:492] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: Bootstrap starting.
I20260924 05:04:00.743724 2633 tablet_bootstrap.cc:492] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: No bootstrap required, opened a new log
I20260924 05:04:00.743772 2633 ts_tablet_manager.cc:1397] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:00.743848 2632 tablet_bootstrap.cc:492] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: No bootstrap required, opened a new log
I20260924 05:04:00.743880 2632 ts_tablet_manager.cc:1397] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:00.743917 2633 raft_consensus.cc:348] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.743984 2633 raft_consensus.cc:374] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.744004 2633 raft_consensus.cc:729] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.743983 2632 raft_consensus.cc:348] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.744035 2632 raft_consensus.cc:374] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.744050 2632 raft_consensus.cc:729] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76e84dca8c3454288c93c1f03c306c5, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.744052 2633 consensus_queue.cc:260] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.744092 2636 tablet_bootstrap.cc:654] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.744089 2632 consensus_queue.cc:260] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.744167 2633 ts_tablet_manager.cc:1428] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.744191 2632 ts_tablet_manager.cc:1428] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.744215 2638 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.744273 2638 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.744287 2637 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.744344 2637 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.744809 2636 tablet_bootstrap.cc:492] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: No bootstrap required, opened a new log
I20260924 05:04:00.744858 2636 ts_tablet_manager.cc:1397] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:00.745000 2636 raft_consensus.cc:348] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.745069 2636 raft_consensus.cc:374] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.745088 2636 raft_consensus.cc:729] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65115da28609498692bd7ef4c3771995, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.745132 2636 consensus_queue.cc:260] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.745214 2636 ts_tablet_manager.cc:1428] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.745275 2640 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.745366 2640 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.782176 2637 raft_consensus.cc:482] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:00.782263 2637 raft_consensus.cc:504] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.782583 2637 leader_election.cc:288] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:00.786191 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.782726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45111 (local address 127.0.98.1:49603)
0924 05:04:00.782839 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.782850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.782909 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.783041 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.783042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.783043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.783154 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.783155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.784027 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.784028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.785296 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.785299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.785938 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.785939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.785986 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.785990 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.786051 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.786093 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":58,"thread_start_us":54,"threads_started":1}
I20260924 05:04:00.786178 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.782781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49603 (local address 127.0.98.2:45111)
0924 05:04:00.782823 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.782825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.782911 (+ 86us) server_negotiation.cc:407] Connection header received
0924 05:04:00.782977 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.782978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.782986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.783032 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.783218 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.783957 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.785374 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.785857 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.785922 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.785932 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.786072 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.786110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:00.786417 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.782781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33359 (local address 127.0.98.3:46855)
0924 05:04:00.782823 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.782825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.782894 (+ 69us) server_negotiation.cc:407] Connection header received
0924 05:04:00.782964 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.782965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.782976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.783032 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.783218 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.783954 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.785528 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.786063 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.786119 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.786129 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.786315 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.786354 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:00.786546 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc8d3488de45d7bd7075085e7a66a6" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995" is_pre_election: true
I20260924 05:04:00.786624 2423 raft_consensus.cc:2454] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 0.
I20260924 05:04:00.786635 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.782726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46855 (local address 127.0.98.1:33359)
0924 05:04:00.782840 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.782850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.782909 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.783041 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.783042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.783045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.783160 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.783162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.784024 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.784025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.785468 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.785469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.786162 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.786163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.786233 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.786238 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.786521 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.786578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":53,"thread_start_us":54,"threads_started":1}
I20260924 05:04:00.786836 2215 leader_election.cc:302] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65115da28609498692bd7ef4c3771995, 77c8531a5c154284b0481b311c7441db; no voters:
I20260924 05:04:00.786895 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc8d3488de45d7bd7075085e7a66a6" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5" is_pre_election: true
I20260924 05:04:00.786985 2637 raft_consensus.cc:2781] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:00.786999 2563 raft_consensus.cc:2454] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 0.
I20260924 05:04:00.787021 2637 raft_consensus.cc:482] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:00.787046 2637 raft_consensus.cc:3037] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.787700 2637 raft_consensus.cc:504] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.787808 2637 leader_election.cc:288] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 election: Requested vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:00.787930 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc8d3488de45d7bd7075085e7a66a6" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995"
I20260924 05:04:00.787946 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "decc8d3488de45d7bd7075085e7a66a6" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5"
I20260924 05:04:00.787999 2423 raft_consensus.cc:3037] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.788002 2563 raft_consensus.cc:3037] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.788707 2423 raft_consensus.cc:2454] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 1.
I20260924 05:04:00.788707 2563 raft_consensus.cc:2454] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 1.
I20260924 05:04:00.788861 2214 leader_election.cc:302] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c8531a5c154284b0481b311c7441db, f76e84dca8c3454288c93c1f03c306c5; no voters:
I20260924 05:04:00.788944 2637 raft_consensus.cc:2781] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:00.789060 2637 raft_consensus.cc:686] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [term 1 LEADER]: Becoming Leader. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Running, Role: LEADER
I20260924 05:04:00.789152 2637 consensus_queue.cc:237] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.789542 2655 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: TxnStatusTablet state changed. Reason: New leader 77c8531a5c154284b0481b311c7441db. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.789641 2655 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:04:00.789834 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 05:04:00.789947 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 05:04:00.789868 2125 catalog_manager.cc:5495] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c8531a5c154284b0481b311c7441db (127.0.98.1). New cstate: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:00.803035 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.800132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.0.1:41956)
0924 05:04:00.800197 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.800210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.800257 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.800451 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.800452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.800453 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:00.800497 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.800499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.801402 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.801403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.802127 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.802129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.802577 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.802577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.802634 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.802652 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.802909 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.802915 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.802941 (+ 26us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.802978 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:04:00.803064 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.800220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.98.1:44247)
0924 05:04:00.800287 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.800289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.800299 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:00.800352 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.800353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.800374 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.800449 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:00.800539 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.801288 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.802196 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.802496 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.802560 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.802738 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.802858 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.802930 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.802946 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.802966 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:04:00.804651 2423 raft_consensus.cc:1261] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Refusing update from remote peer 77c8531a5c154284b0481b311c7441db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:00.804733 2563 raft_consensus.cc:1261] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Refusing update from remote peer 77c8531a5c154284b0481b311c7441db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:00.804987 2655 consensus_queue.cc:1035] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [LEADER]: Connected to new peer: Peer: permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:00.805107 2637 consensus_queue.cc:1035] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:00.806154 2638 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: TxnStatusTablet state changed. Reason: New leader 77c8531a5c154284b0481b311c7441db. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.806241 2638 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.806293 2640 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: TxnStatusTablet state changed. Reason: New leader 77c8531a5c154284b0481b311c7441db. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.806380 2640 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.807080 2655 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.807159 2655 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:04:00.807232 2660 mvcc.cc:205] Tried to move back new op lower bound from 7332766682333528064 to 7332766682272923648. Current Snapshot: MvccSnapshot[applied={T|T < 7332766682333528064}]
I20260924 05:04:00.807263 2638 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.807329 2638 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P f76e84dca8c3454288c93c1f03c306c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.807529 2661 mvcc.cc:205] Tried to move back new op lower bound from 7332766682333528064 to 7332766682272923648. Current Snapshot: MvccSnapshot[applied={T|T < 7332766682333528064}]
I20260924 05:04:00.807673 2667 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.807775 2667 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 77c8531a5c154284b0481b311c7441db: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:04:00.808514 2640 tablet_replica.cc:439] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } } }
I20260924 05:04:00.808591 2640 tablet_replica.cc:442] T decc8d3488de45d7bd7075085e7a66a6 P 65115da28609498692bd7ef4c3771995: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:00.809440 2662 mvcc.cc:205] Tried to move back new op lower bound from 7332766682333528064 to 7332766682272923648. Current Snapshot: MvccSnapshot[applied={T|T < 7332766682333528064}]
I20260924 05:04:00.812170 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.809751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.0.98.3:46855)
0924 05:04:00.809785 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.809786 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.809793 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:00.809819 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.809819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.809835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.809897 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:00.810044 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.810495 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.811559 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.811744 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.811789 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.811928 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.812005 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.812052 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.812105 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.812122 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:00.816674 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.814247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.98.3:51021)
0924 05:04:00.814305 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.814316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.814367 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.814489 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.814489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.814490 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:00.814534 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.814535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.815335 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.815336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.815944 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.815946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.816373 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.816374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.816415 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.816427 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.816572 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.816573 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.816600 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.816627 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:04:00.816687 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.814318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51021 (local address 127.0.98.1:44247)
0924 05:04:00.814353 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.814355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.814362 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:00.814425 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.814425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.814435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.814487 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:00.814571 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.815271 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.816014 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.816317 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.816378 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.816494 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.816523 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.816562 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.816605 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.816621 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:00.818928 2283 raft_consensus.cc:1261] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Refusing update from remote peer f76e84dca8c3454288c93c1f03c306c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:00.819097 2423 raft_consensus.cc:1261] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Refusing update from remote peer f76e84dca8c3454288c93c1f03c306c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:00.819192 2642 consensus_queue.cc:1035] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:00.819296 2638 consensus_queue.cc:1035] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:00.828097 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.825366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47823 (local address 127.0.98.3:46855)
0924 05:04:00.825400 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.825402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.825410 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:00.825467 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.825467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.825478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.825530 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:00.825653 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.826426 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.827405 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.827709 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.827779 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.827897 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.827927 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.827968 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.828003 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.828021 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:00.828117 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.825190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46855 (local address 127.0.98.1:47823)
0924 05:04:00.825358 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.825370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.825415 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.825531 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.825532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.825533 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:00.825598 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.825600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.826493 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.826493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.827300 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.827303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.827775 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.827776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.827823 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.827836 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:00.827971 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:00.827972 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.828000 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.828033 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260924 05:04:00.836659 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:00.836771 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:00.837450 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Sending DeleteTablet for 3 replicas of tablet 88e5967d3ae3451e8fc66384cb86cca5
I20260924 05:04:00.837809 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 88e5967d3ae3451e8fc66384cb86cca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260924 05:04:00.838029 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 88e5967d3ae3451e8fc66384cb86cca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41954
I20260924 05:04:00.838269 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 88e5967d3ae3451e8fc66384cb86cca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35516
I20260924 05:04:00.838511 2681 tablet_replica.cc:330] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: stopping tablet replica
I20260924 05:04:00.838519 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 05:04:00.838644 2681 raft_consensus.cc:2229] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 1 LEADER]: Raft consensus shutting down.
W20260924 05:04:00.838666 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:00.838750 2681 raft_consensus.cc:2258] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:00.839213 2681 ts_tablet_manager.cc:1905] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:00.840013 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 979a709fa19f414eba25f11140dc0b36 (DEFAULT_TABLE table=test-table [id=9d207227524c4c27bacf17635ef46393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:00.840134 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979a709fa19f414eba25f11140dc0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.840992 2681 ts_tablet_manager.cc:1918] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:00.841050 2681 log.cc:1196] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/88e5967d3ae3451e8fc66384cb86cca5
I20260924 05:04:00.841280 2681 ts_tablet_manager.cc:1939] T 88e5967d3ae3451e8fc66384cb86cca5 P f76e84dca8c3454288c93c1f03c306c5: Deleting consensus metadata
I20260924 05:04:00.841495 2636 tablet_bootstrap.cc:492] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Bootstrap starting.
I20260924 05:04:00.841850 2636 tablet_bootstrap.cc:654] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.842336 2110 catalog_manager.cc:4841] TS f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855): tablet 88e5967d3ae3451e8fc66384cb86cca5 (table test-table [id=f067b38ebe51418d82ba7bbe07c7da52]) successfully deleted
I20260924 05:04:00.842363 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 979a709fa19f414eba25f11140dc0b36 (DEFAULT_TABLE table=test-table [id=9d207227524c4c27bacf17635ef46393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:00.842442 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979a709fa19f414eba25f11140dc0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.842501 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 979a709fa19f414eba25f11140dc0b36 (DEFAULT_TABLE table=test-table [id=9d207227524c4c27bacf17635ef46393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:00.842607 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 979a709fa19f414eba25f11140dc0b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:00.843518 2632 tablet_bootstrap.cc:492] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Bootstrap starting.
I20260924 05:04:00.843981 2632 tablet_bootstrap.cc:654] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.844282 2633 tablet_bootstrap.cc:492] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Bootstrap starting.
I20260924 05:04:00.844485 2679 tablet_replica.cc:330] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: stopping tablet replica
I20260924 05:04:00.844602 2679 raft_consensus.cc:2229] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:00.844655 2633 tablet_bootstrap.cc:654] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:00.844663 2679 raft_consensus.cc:2258] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:00.844825 2680 tablet_replica.cc:330] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: stopping tablet replica
I20260924 05:04:00.844918 2680 raft_consensus.cc:2229] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:00.844962 2680 raft_consensus.cc:2258] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:00.845338 2633 tablet_bootstrap.cc:492] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: No bootstrap required, opened a new log
I20260924 05:04:00.845381 2633 ts_tablet_manager.cc:1397] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:00.845402 2680 ts_tablet_manager.cc:1905] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:00.845510 2633 raft_consensus.cc:348] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.845587 2633 raft_consensus.cc:374] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.845606 2633 raft_consensus.cc:729] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.845646 2633 consensus_queue.cc:260] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.845759 2633 ts_tablet_manager.cc:1428] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.846133 2636 tablet_bootstrap.cc:492] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: No bootstrap required, opened a new log
I20260924 05:04:00.846181 2636 ts_tablet_manager.cc:1397] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 05:04:00.846232 2632 tablet_bootstrap.cc:492] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: No bootstrap required, opened a new log
I20260924 05:04:00.846267 2632 ts_tablet_manager.cc:1397] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:04:00.846313 2636 raft_consensus.cc:348] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.846374 2636 raft_consensus.cc:374] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.846390 2636 raft_consensus.cc:729] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65115da28609498692bd7ef4c3771995, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.846391 2632 raft_consensus.cc:348] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.846446 2632 raft_consensus.cc:374] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:00.846428 2636 consensus_queue.cc:260] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.846464 2632 raft_consensus.cc:729] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76e84dca8c3454288c93c1f03c306c5, State: Initialized, Role: FOLLOWER
I20260924 05:04:00.846503 2636 ts_tablet_manager.cc:1428] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.846496 2632 consensus_queue.cc:260] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.846567 2632 ts_tablet_manager.cc:1428] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:00.846881 2680 ts_tablet_manager.cc:1918] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:00.846972 2680 log.cc:1196] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/88e5967d3ae3451e8fc66384cb86cca5
I20260924 05:04:00.847244 2680 ts_tablet_manager.cc:1939] T 88e5967d3ae3451e8fc66384cb86cca5 P 77c8531a5c154284b0481b311c7441db: Deleting consensus metadata
I20260924 05:04:00.847597 2110 catalog_manager.cc:4841] TS 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247): tablet 88e5967d3ae3451e8fc66384cb86cca5 (table test-table [id=f067b38ebe51418d82ba7bbe07c7da52]) successfully deleted
I20260924 05:04:00.850019 2679 ts_tablet_manager.cc:1905] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:00.851207 2679 ts_tablet_manager.cc:1918] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:00.851313 2679 log.cc:1196] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/88e5967d3ae3451e8fc66384cb86cca5
I20260924 05:04:00.851562 2679 ts_tablet_manager.cc:1939] T 88e5967d3ae3451e8fc66384cb86cca5 P 65115da28609498692bd7ef4c3771995: Deleting consensus metadata
I20260924 05:04:00.851958 2111 catalog_manager.cc:4841] TS 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111): tablet 88e5967d3ae3451e8fc66384cb86cca5 (table test-table [id=f067b38ebe51418d82ba7bbe07c7da52]) successfully deleted
I20260924 05:04:00.993022 2640 raft_consensus.cc:482] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:00.993109 2640 raft_consensus.cc:504] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.993408 2640 leader_election.cc:288] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:00.996202 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.993881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46043 (local address 127.0.98.3:46855)
0924 05:04:00.993921 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.993923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.993954 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:04:00.994012 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.994012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.994021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.994065 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.994188 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.994683 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.995577 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.995904 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.995961 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.995972 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.996112 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.996138 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:00.996207 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.993696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45915 (local address 127.0.98.1:44247)
0924 05:04:00.993771 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:00.993773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:00.993789 (+ 16us) server_negotiation.cc:407] Connection header received
0924 05:04:00.993891 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:00.993892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:00.993902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:00.993930 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:00.994180 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.994654 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.995557 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.995906 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.995962 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.995971 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:00.996107 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 05:04:00.996136 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 05:04:00.996227 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.993581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.98.2:45915)
0924 05:04:00.993705 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.993745 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.993806 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.993993 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.993994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.993997 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.994066 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.994068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.994730 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.994731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.995473 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.995475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.995976 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.995977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.996044 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.996049 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.996092 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.996131 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260924 05:04:00.996233 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:00.993785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46855 (local address 127.0.98.2:46043)
0924 05:04:00.993894 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:00.993903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:00.993960 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:00.994070 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:00.994071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:00.994073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:00.994130 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.994132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.994765 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.994766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.995486 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:00.995488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:00.995979 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:00.995980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:00.996044 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:00.996047 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:00.996098 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:04:00.996138 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":44,"threads_started":1}
I20260924 05:04:00.996462 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a709fa19f414eba25f11140dc0b36" candidate_uuid: "65115da28609498692bd7ef4c3771995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c8531a5c154284b0481b311c7441db" is_pre_election: true
I20260924 05:04:00.996551 2283 raft_consensus.cc:2454] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65115da28609498692bd7ef4c3771995 in term 0.
I20260924 05:04:00.996709 2354 leader_election.cc:302] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65115da28609498692bd7ef4c3771995, 77c8531a5c154284b0481b311c7441db; no voters:
I20260924 05:04:00.996697 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a709fa19f414eba25f11140dc0b36" candidate_uuid: "65115da28609498692bd7ef4c3771995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5" is_pre_election: true
I20260924 05:04:00.996771 2563 raft_consensus.cc:2454] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65115da28609498692bd7ef4c3771995 in term 0.
I20260924 05:04:00.996819 2640 raft_consensus.cc:2781] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:00.996874 2640 raft_consensus.cc:482] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:00.996896 2640 raft_consensus.cc:3037] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.997491 2640 raft_consensus.cc:504] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.997596 2640 leader_election.cc:288] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [CANDIDATE]: Term 1 election: Requested vote from peers 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:00.997766 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a709fa19f414eba25f11140dc0b36" candidate_uuid: "65115da28609498692bd7ef4c3771995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c8531a5c154284b0481b311c7441db"
I20260924 05:04:00.997785 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "979a709fa19f414eba25f11140dc0b36" candidate_uuid: "65115da28609498692bd7ef4c3771995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5"
I20260924 05:04:00.997843 2283 raft_consensus.cc:3037] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.997843 2563 raft_consensus.cc:3037] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:00.998365 2563 raft_consensus.cc:2454] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65115da28609498692bd7ef4c3771995 in term 1.
I20260924 05:04:00.998445 2283 raft_consensus.cc:2454] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65115da28609498692bd7ef4c3771995 in term 1.
I20260924 05:04:00.998524 2354 leader_election.cc:302] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65115da28609498692bd7ef4c3771995, f76e84dca8c3454288c93c1f03c306c5; no voters:
I20260924 05:04:00.998625 2640 raft_consensus.cc:2781] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:00.998740 2640 raft_consensus.cc:686] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 1 LEADER]: Becoming Leader. State: Replica: 65115da28609498692bd7ef4c3771995, State: Running, Role: LEADER
I20260924 05:04:00.998831 2640 consensus_queue.cc:237] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:00.999420 2126 catalog_manager.cc:5495] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65115da28609498692bd7ef4c3771995 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "65115da28609498692bd7ef4c3771995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:01.030262 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:01.028307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.0.98.2:45111)
0924 05:04:01.028343 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:01.028344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:01.028350 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:01.028382 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:01.028383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:01.028401 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:01.028461 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:01.028541 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.028995 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.029658 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.029846 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.029890 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:01.030001 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:01.030104 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:01.030149 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:01.030167 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:04:01.030182 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:01.034744 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:01.032570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42473 (local address 127.0.98.1:44247)
0924 05:04:01.032628 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:01.032630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:01.032679 (+ 49us) server_negotiation.cc:407] Connection header received
0924 05:04:01.032749 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:01.032750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:01.032759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:01.032806 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:01.032925 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.033381 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.034169 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.034349 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.034399 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:01.034529 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:01.034560 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:01.034599 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:01.034647 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 05:04:01.034666 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:04:01.034780 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:01.032485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.98.2:42473)
0924 05:04:01.032612 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:01.032624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:01.032695 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:01.032811 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:01.032812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:01.032813 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:01.032873 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:01.032875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.033447 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.033448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:01.034083 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:01.034085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.034401 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.034402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:01.034451 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:01.034464 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:01.034606 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:01.034608 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:01.034646 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 05:04:01.034691 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260924 05:04:01.036794 2563 raft_consensus.cc:1261] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Refusing update from remote peer 65115da28609498692bd7ef4c3771995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:01.036823 2283 raft_consensus.cc:1261] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Refusing update from remote peer 65115da28609498692bd7ef4c3771995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:01.037045 2640 consensus_queue.cc:1035] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:01.037374 2687 consensus_queue.cc:1035] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:01.043839 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:01.041780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57443 (local address 127.0.98.2:45111)
0924 05:04:01.041824 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:01.041826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:01.041865 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:04:01.041923 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:01.041924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:01.041933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:01.041982 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:01.042075 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.042531 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.043277 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.043472 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.043524 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:01.043636 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:01.043668 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:01.043710 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:01.043755 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 05:04:01.043772 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:01.043859 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:01.041758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45111 (local address 127.0.98.1:57443)
0924 05:04:01.041821 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:01.041832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:01.041875 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:01.041985 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:01.041986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:01.041987 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:01.042027 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:01.042029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.042594 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.042595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:01.043208 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:01.043210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:01.043524 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:01.043525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:01.043576 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:01.043582 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:01.043714 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:01.043715 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:04:01.043754 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 05:04:01.043796 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 05:04:01.044183 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 05:04:02.561856 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:02.561949 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:02.562636 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Sending DeleteTablet for 3 replicas of tablet 979a709fa19f414eba25f11140dc0b36
I20260924 05:04:02.562886 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 979a709fa19f414eba25f11140dc0b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41954
I20260924 05:04:02.562896 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 979a709fa19f414eba25f11140dc0b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35516
I20260924 05:04:02.563050 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 979a709fa19f414eba25f11140dc0b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260924 05:04:02.563197 2718 tablet_replica.cc:330] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: stopping tablet replica
I20260924 05:04:02.563223 2717 tablet_replica.cc:330] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: stopping tablet replica
I20260924 05:04:02.563241 2718 raft_consensus.cc:2229] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:02.563271 2717 raft_consensus.cc:2229] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:02.563298 2718 raft_consensus.cc:2258] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:02.563302 2717 raft_consensus.cc:2258] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:02.563398 2716 tablet_replica.cc:330] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: stopping tablet replica
I20260924 05:04:02.563442 2716 raft_consensus.cc:2229] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:02.563467 2716 raft_consensus.cc:2258] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:02.563541 2718 ts_tablet_manager.cc:1905] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:02.563621 2717 ts_tablet_manager.cc:1905] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:02.564937 2718 ts_tablet_manager.cc:1918] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:04:02.564990 2718 log.cc:1196] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/979a709fa19f414eba25f11140dc0b36
I20260924 05:04:02.565132 2717 ts_tablet_manager.cc:1918] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:04:02.565191 2718 ts_tablet_manager.cc:1939] T 979a709fa19f414eba25f11140dc0b36 P 65115da28609498692bd7ef4c3771995: Deleting consensus metadata
I20260924 05:04:02.565191 2717 log.cc:1196] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/979a709fa19f414eba25f11140dc0b36
I20260924 05:04:02.565380 2717 ts_tablet_manager.cc:1939] T 979a709fa19f414eba25f11140dc0b36 P f76e84dca8c3454288c93c1f03c306c5: Deleting consensus metadata
I20260924 05:04:02.565407 2111 catalog_manager.cc:4841] TS 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111): tablet 979a709fa19f414eba25f11140dc0b36 (table test-table [id=9d207227524c4c27bacf17635ef46393]) successfully deleted
I20260924 05:04:02.565570 2716 ts_tablet_manager.cc:1905] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:02.565682 2110 catalog_manager.cc:4841] TS f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855): tablet 979a709fa19f414eba25f11140dc0b36 (table test-table [id=9d207227524c4c27bacf17635ef46393]) successfully deleted
I20260924 05:04:02.566686 2716 ts_tablet_manager.cc:1918] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:04:02.566740 2716 log.cc:1196] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/979a709fa19f414eba25f11140dc0b36
I20260924 05:04:02.566896 2716 ts_tablet_manager.cc:1939] T 979a709fa19f414eba25f11140dc0b36 P 77c8531a5c154284b0481b311c7441db: Deleting consensus metadata
I20260924 05:04:02.567184 2110 catalog_manager.cc:4841] TS 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247): tablet 979a709fa19f414eba25f11140dc0b36 (table test-table [id=9d207227524c4c27bacf17635ef46393]) successfully deleted
Sep 24 05:04:02 dist-test-slave-m01t krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:04:02 dist-test-slave-m01t krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:02.584977 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:02.578920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.0.98.62:36217)
0924 05:04:02.579043 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:02.579044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:02.579053 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:02.579092 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:02.579093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:02.579101 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:02.579153 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:02.579626 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.580113 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.580778 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.580957 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.580996 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:02.582986 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:02.582988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:02.582988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:02.582994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:02.584246 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:02.584484 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:02.584484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:02.584485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:02.584529 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:02.584640 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:02.584641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:02.584641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:02.584729 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:02.584793 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:02.584859 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:04:02.584915 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260924 05:04:02.586175 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:02.586306 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:02.587401 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 4947ca0f408f4370a2d54dc94feff868 (DEFAULT_TABLE table=test-table [id=e8a5bc1cb52445a5a79f73cf2334daed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:02.587502 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4947ca0f408f4370a2d54dc94feff868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:02.588336 2727 tablet_bootstrap.cc:492] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Bootstrap starting.
I20260924 05:04:02.588357 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 4947ca0f408f4370a2d54dc94feff868 (DEFAULT_TABLE table=test-table [id=e8a5bc1cb52445a5a79f73cf2334daed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:02.588467 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4947ca0f408f4370a2d54dc94feff868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:02.588696 2727 tablet_bootstrap.cc:654] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:02.589257 2727 tablet_bootstrap.cc:492] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: No bootstrap required, opened a new log
I20260924 05:04:02.589298 2727 ts_tablet_manager.cc:1397] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:02.589387 2727 raft_consensus.cc:348] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.589457 2727 raft_consensus.cc:374] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:02.589469 2727 raft_consensus.cc:729] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76e84dca8c3454288c93c1f03c306c5, State: Initialized, Role: FOLLOWER
I20260924 05:04:02.589521 2727 consensus_queue.cc:260] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.589663 2729 tablet_bootstrap.cc:492] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Bootstrap starting.
I20260924 05:04:02.589735 2727 ts_tablet_manager.cc:1428] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:02.589702 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 4947ca0f408f4370a2d54dc94feff868 (DEFAULT_TABLE table=test-table [id=e8a5bc1cb52445a5a79f73cf2334daed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:02.589831 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4947ca0f408f4370a2d54dc94feff868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:02.590021 2729 tablet_bootstrap.cc:654] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:02.591114 2732 tablet_bootstrap.cc:492] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Bootstrap starting.
I20260924 05:04:02.591432 2732 tablet_bootstrap.cc:654] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:02.591506 2729 tablet_bootstrap.cc:492] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: No bootstrap required, opened a new log
I20260924 05:04:02.591559 2729 ts_tablet_manager.cc:1397] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:02.591670 2729 raft_consensus.cc:348] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.591744 2729 raft_consensus.cc:374] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:02.591759 2729 raft_consensus.cc:729] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65115da28609498692bd7ef4c3771995, State: Initialized, Role: FOLLOWER
I20260924 05:04:02.591794 2729 consensus_queue.cc:260] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.591900 2729 ts_tablet_manager.cc:1428] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:02.591981 2732 tablet_bootstrap.cc:492] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: No bootstrap required, opened a new log
I20260924 05:04:02.592031 2732 ts_tablet_manager.cc:1397] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:02.592130 2732 raft_consensus.cc:348] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.592195 2732 raft_consensus.cc:374] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:02.592213 2732 raft_consensus.cc:729] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Initialized, Role: FOLLOWER
I20260924 05:04:02.592250 2732 consensus_queue.cc:260] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.592351 2732 ts_tablet_manager.cc:1428] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:02.804055 2698 raft_consensus.cc:482] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:02.804150 2698 raft_consensus.cc:504] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.804333 2698 leader_election.cc:288] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:02.804512 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4947ca0f408f4370a2d54dc94feff868" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995" is_pre_election: true
I20260924 05:04:02.804518 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4947ca0f408f4370a2d54dc94feff868" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5" is_pre_election: true
I20260924 05:04:02.804596 2423 raft_consensus.cc:2454] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 0.
I20260924 05:04:02.804621 2563 raft_consensus.cc:2454] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 0.
I20260924 05:04:02.804792 2215 leader_election.cc:302] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65115da28609498692bd7ef4c3771995, 77c8531a5c154284b0481b311c7441db; no voters:
I20260924 05:04:02.804893 2698 raft_consensus.cc:2781] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:02.804929 2698 raft_consensus.cc:482] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:02.804942 2698 raft_consensus.cc:3037] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:02.805581 2698 raft_consensus.cc:504] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.805691 2698 leader_election.cc:288] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 election: Requested vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:02.805850 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4947ca0f408f4370a2d54dc94feff868" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5"
I20260924 05:04:02.805858 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4947ca0f408f4370a2d54dc94feff868" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995"
I20260924 05:04:02.805917 2563 raft_consensus.cc:3037] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:02.805922 2423 raft_consensus.cc:3037] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:02.806461 2423 raft_consensus.cc:2454] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 1.
I20260924 05:04:02.806473 2563 raft_consensus.cc:2454] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 1.
I20260924 05:04:02.806629 2214 leader_election.cc:302] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c8531a5c154284b0481b311c7441db, f76e84dca8c3454288c93c1f03c306c5; no voters:
I20260924 05:04:02.806726 2698 raft_consensus.cc:2781] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:02.806788 2698 raft_consensus.cc:686] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 1 LEADER]: Becoming Leader. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Running, Role: LEADER
I20260924 05:04:02.806838 2698 consensus_queue.cc:237] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.807406 2126 catalog_manager.cc:5495] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c8531a5c154284b0481b311c7441db (127.0.98.1). New cstate: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:02.815894 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:02.813983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.98.1:44247)
0924 05:04:02.814100 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:02.814102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:02.814109 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:02.814126 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:02.814126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:02.814135 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:02.814190 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:02.814310 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.814773 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.815418 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.815597 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.815636 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:02.815731 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:02.815755 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:02.815783 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:02.815816 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 05:04:02.815829 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260924 05:04:02.819649 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:02.817740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45889 (local address 127.0.98.1:44247)
0924 05:04:02.817769 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:02.817771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:02.817808 (+ 37us) server_negotiation.cc:407] Connection header received
0924 05:04:02.817862 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:02.817863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:02.817871 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:02.817912 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:02.818006 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.818422 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.819082 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.819267 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.819312 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:02.819434 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:02.819467 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:02.819523 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:02.819565 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 05:04:02.819584 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:02.819657 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:02.817632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.98.1:45889)
0924 05:04:02.817761 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:02.817771 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:02.817822 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:02.817918 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:02.817919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:02.817920 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:02.817955 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:02.817956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.818482 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.818483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:02.819009 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:02.819011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:02.819328 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:02.819329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:02.819372 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:02.819377 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:02.819522 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:02.819523 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:04:02.819552 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 05:04:02.819596 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260924 05:04:02.821163 2563 raft_consensus.cc:1261] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Refusing update from remote peer 77c8531a5c154284b0481b311c7441db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:02.821193 2423 raft_consensus.cc:1261] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Refusing update from remote peer 77c8531a5c154284b0481b311c7441db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:02.821357 2703 consensus_queue.cc:1035] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [LEADER]: Connected to new peer: Peer: permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:02.821455 2667 consensus_queue.cc:1035] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:02.830852 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:02.830952 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:02.831526 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Sending DeleteTablet for 3 replicas of tablet 4947ca0f408f4370a2d54dc94feff868
I20260924 05:04:02.831746 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 4947ca0f408f4370a2d54dc94feff868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41954
I20260924 05:04:02.831787 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 4947ca0f408f4370a2d54dc94feff868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260924 05:04:02.831813 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 4947ca0f408f4370a2d54dc94feff868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35516
I20260924 05:04:02.831838 2716 tablet_replica.cc:330] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: stopping tablet replica
I20260924 05:04:02.831876 2718 tablet_replica.cc:330] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: stopping tablet replica
I20260924 05:04:02.831887 2716 raft_consensus.cc:2229] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:02.831893 2717 tablet_replica.cc:330] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: stopping tablet replica
I20260924 05:04:02.831918 2718 raft_consensus.cc:2229] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:02.831935 2717 raft_consensus.cc:2229] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:02.831952 2718 raft_consensus.cc:2258] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:02.831964 2716 raft_consensus.cc:2258] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:02.831970 2717 raft_consensus.cc:2258] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:02.832249 2718 ts_tablet_manager.cc:1905] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:02.832378 2716 ts_tablet_manager.cc:1905] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:02.832518 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:02.832648 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:02.833927 2716 ts_tablet_manager.cc:1918] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:02.833997 2716 log.cc:1196] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/4947ca0f408f4370a2d54dc94feff868
I20260924 05:04:02.834194 2716 ts_tablet_manager.cc:1939] T 4947ca0f408f4370a2d54dc94feff868 P 77c8531a5c154284b0481b311c7441db: Deleting consensus metadata
I20260924 05:04:02.834349 2403 tablet_service.cc:1467] Processing CreateTablet for tablet cca6cf3325f8471da1d6d63c4443b9fc (DEFAULT_TABLE table=test-table [id=2130ac1882064274a277ab260436d60d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:02.834350 2543 tablet_service.cc:1467] Processing CreateTablet for tablet cca6cf3325f8471da1d6d63c4443b9fc (DEFAULT_TABLE table=test-table [id=2130ac1882064274a277ab260436d60d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:02.834455 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca6cf3325f8471da1d6d63c4443b9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:02.834595 2717 ts_tablet_manager.cc:1905] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:02.834659 2718 ts_tablet_manager.cc:1918] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:02.834700 2718 log.cc:1196] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/4947ca0f408f4370a2d54dc94feff868
I20260924 05:04:02.834867 2718 ts_tablet_manager.cc:1939] T 4947ca0f408f4370a2d54dc94feff868 P 65115da28609498692bd7ef4c3771995: Deleting consensus metadata
I20260924 05:04:02.834971 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca6cf3325f8471da1d6d63c4443b9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:02.835669 2727 tablet_bootstrap.cc:492] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Bootstrap starting.
I20260924 05:04:02.835969 2727 tablet_bootstrap.cc:654] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:02.836019 2717 ts_tablet_manager.cc:1918] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:02.836064 2717 log.cc:1196] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/4947ca0f408f4370a2d54dc94feff868
I20260924 05:04:02.836097 2729 tablet_bootstrap.cc:492] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Bootstrap starting.
I20260924 05:04:02.836241 2717 ts_tablet_manager.cc:1939] T 4947ca0f408f4370a2d54dc94feff868 P f76e84dca8c3454288c93c1f03c306c5: Deleting consensus metadata
I20260924 05:04:02.836380 2729 tablet_bootstrap.cc:654] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:02.836679 2263 tablet_service.cc:1467] Processing CreateTablet for tablet cca6cf3325f8471da1d6d63c4443b9fc (DEFAULT_TABLE table=test-table [id=2130ac1882064274a277ab260436d60d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:02.836740 2110 catalog_manager.cc:4841] TS 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247): tablet 4947ca0f408f4370a2d54dc94feff868 (table test-table [id=e8a5bc1cb52445a5a79f73cf2334daed]) successfully deleted
I20260924 05:04:02.836776 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca6cf3325f8471da1d6d63c4443b9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:02.836805 2110 catalog_manager.cc:4841] TS f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855): tablet 4947ca0f408f4370a2d54dc94feff868 (table test-table [id=e8a5bc1cb52445a5a79f73cf2334daed]) successfully deleted
I20260924 05:04:02.836974 2729 tablet_bootstrap.cc:492] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: No bootstrap required, opened a new log
I20260924 05:04:02.837008 2729 ts_tablet_manager.cc:1397] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:02.837103 2729 raft_consensus.cc:348] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.837155 2729 raft_consensus.cc:374] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:02.837172 2729 raft_consensus.cc:729] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65115da28609498692bd7ef4c3771995, State: Initialized, Role: FOLLOWER
I20260924 05:04:02.837217 2729 consensus_queue.cc:260] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.837298 2729 ts_tablet_manager.cc:1428] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:02.837996 2732 tablet_bootstrap.cc:492] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db: Bootstrap starting.
I20260924 05:04:02.838305 2732 tablet_bootstrap.cc:654] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:02.838426 2111 catalog_manager.cc:4841] TS 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111): tablet 4947ca0f408f4370a2d54dc94feff868 (table test-table [id=e8a5bc1cb52445a5a79f73cf2334daed]) successfully deleted
I20260924 05:04:02.838631 2727 tablet_bootstrap.cc:492] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: No bootstrap required, opened a new log
I20260924 05:04:02.838671 2727 ts_tablet_manager.cc:1397] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 05:04:02.838766 2727 raft_consensus.cc:348] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.838824 2727 raft_consensus.cc:374] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:02.838835 2727 raft_consensus.cc:729] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76e84dca8c3454288c93c1f03c306c5, State: Initialized, Role: FOLLOWER
I20260924 05:04:02.838865 2727 consensus_queue.cc:260] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.838970 2727 ts_tablet_manager.cc:1428] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:02.839416 2732 tablet_bootstrap.cc:492] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db: No bootstrap required, opened a new log
I20260924 05:04:02.839465 2732 ts_tablet_manager.cc:1397] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:02.839583 2732 raft_consensus.cc:348] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.839643 2732 raft_consensus.cc:374] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:02.839676 2732 raft_consensus.cc:729] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Initialized, Role: FOLLOWER
I20260924 05:04:02.839710 2732 consensus_queue.cc:260] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.839784 2732 ts_tablet_manager.cc:1428] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:02.843051 2703 raft_consensus.cc:482] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:02.843108 2703 raft_consensus.cc:504] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.843225 2703 leader_election.cc:288] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:02.843333 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca6cf3325f8471da1d6d63c4443b9fc" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5" is_pre_election: true
I20260924 05:04:02.843345 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca6cf3325f8471da1d6d63c4443b9fc" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995" is_pre_election: true
I20260924 05:04:02.843406 2422 raft_consensus.cc:2454] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 0.
I20260924 05:04:02.843407 2563 raft_consensus.cc:2454] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 0.
I20260924 05:04:02.843532 2214 leader_election.cc:302] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c8531a5c154284b0481b311c7441db, f76e84dca8c3454288c93c1f03c306c5; no voters:
I20260924 05:04:02.843618 2703 raft_consensus.cc:2781] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:02.843664 2703 raft_consensus.cc:482] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:02.843691 2703 raft_consensus.cc:3037] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:02.844153 2703 raft_consensus.cc:504] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.844262 2703 leader_election.cc:288] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 election: Requested vote from peers 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111), f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855)
I20260924 05:04:02.844379 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca6cf3325f8471da1d6d63c4443b9fc" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65115da28609498692bd7ef4c3771995"
I20260924 05:04:02.844448 2422 raft_consensus.cc:3037] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:02.844431 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca6cf3325f8471da1d6d63c4443b9fc" candidate_uuid: "77c8531a5c154284b0481b311c7441db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76e84dca8c3454288c93c1f03c306c5"
I20260924 05:04:02.844493 2563 raft_consensus.cc:3037] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:02.844956 2422 raft_consensus.cc:2454] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 1.
I20260924 05:04:02.845109 2215 leader_election.cc:302] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65115da28609498692bd7ef4c3771995, 77c8531a5c154284b0481b311c7441db; no voters:
I20260924 05:04:02.845116 2563 raft_consensus.cc:2454] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c8531a5c154284b0481b311c7441db in term 1.
I20260924 05:04:02.845264 2703 raft_consensus.cc:2781] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:02.845309 2703 raft_consensus.cc:686] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [term 1 LEADER]: Becoming Leader. State: Replica: 77c8531a5c154284b0481b311c7441db, State: Running, Role: LEADER
I20260924 05:04:02.845345 2703 consensus_queue.cc:237] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } }
I20260924 05:04:02.845836 2124 catalog_manager.cc:5495] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c8531a5c154284b0481b311c7441db (127.0.98.1). New cstate: current_term: 1 leader_uuid: "77c8531a5c154284b0481b311c7441db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77c8531a5c154284b0481b311c7441db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:02.853073 2563 raft_consensus.cc:1261] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Refusing update from remote peer 77c8531a5c154284b0481b311c7441db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:02.853066 2422 raft_consensus.cc:1261] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Refusing update from remote peer 77c8531a5c154284b0481b311c7441db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:02.853237 2703 consensus_queue.cc:1035] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76e84dca8c3454288c93c1f03c306c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:02.853327 2667 consensus_queue.cc:1035] T cca6cf3325f8471da1d6d63c4443b9fc P 77c8531a5c154284b0481b311c7441db [LEADER]: Connected to new peer: Peer: permanent_uuid: "65115da28609498692bd7ef4c3771995" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:04:02.857118 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 05:04:04.373175 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:04.373274 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:04.373999 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 007d5f161bdf48e091026c45fd241ca9: Sending DeleteTablet for 3 replicas of tablet cca6cf3325f8471da1d6d63c4443b9fc
I20260924 05:04:04.374207 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet cca6cf3325f8471da1d6d63c4443b9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41954
I20260924 05:04:04.374239 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet cca6cf3325f8471da1d6d63c4443b9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260924 05:04:04.374256 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet cca6cf3325f8471da1d6d63c4443b9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35516
I20260924 05:04:04.374342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 2198
I20260924 05:04:04.377805 2770 tablet_replica.cc:330] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: stopping tablet replica
I20260924 05:04:04.377875 2770 raft_consensus.cc:2229] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:04.377914 2770 raft_consensus.cc:2258] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:04.378006 2769 tablet_replica.cc:330] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: stopping tablet replica
I20260924 05:04:04.378118 2769 raft_consensus.cc:2229] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:04.378144 2770 ts_tablet_manager.cc:1905] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:04.378170 2769 raft_consensus.cc:2258] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:04.378401 2769 ts_tablet_manager.cc:1905] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:04.379515 2770 ts_tablet_manager.cc:1918] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:04:04.379567 2770 log.cc:1196] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/cca6cf3325f8471da1d6d63c4443b9fc
I20260924 05:04:04.379745 2770 ts_tablet_manager.cc:1939] T cca6cf3325f8471da1d6d63c4443b9fc P f76e84dca8c3454288c93c1f03c306c5: Deleting consensus metadata
I20260924 05:04:04.379801 2769 ts_tablet_manager.cc:1918] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:04:04.379859 2769 log.cc:1196] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/cca6cf3325f8471da1d6d63c4443b9fc
I20260924 05:04:04.380012 2110 catalog_manager.cc:4841] TS f76e84dca8c3454288c93c1f03c306c5 (127.0.98.3:46855): tablet cca6cf3325f8471da1d6d63c4443b9fc (table test-table [id=2130ac1882064274a277ab260436d60d]) successfully deleted
I20260924 05:04:04.380040 2769 ts_tablet_manager.cc:1939] T cca6cf3325f8471da1d6d63c4443b9fc P 65115da28609498692bd7ef4c3771995: Deleting consensus metadata
I20260924 05:04:04.380312 2111 catalog_manager.cc:4841] TS 65115da28609498692bd7ef4c3771995 (127.0.98.2:45111): tablet cca6cf3325f8471da1d6d63c4443b9fc (table test-table [id=2130ac1882064274a277ab260436d60d]) successfully deleted
W20260924 05:04:04.381306 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44247 (error 108)
I20260924 05:04:04.381454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 2338
I20260924 05:04:04.382702 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:04.382515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44247 (local address 127.0.0.1:56574)
0924 05:04:04.382624 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:04.382671 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
W20260924 05:04:04.382813 2110 catalog_manager.cc:4584] TS 77c8531a5c154284b0481b311c7441db (127.0.98.1:44247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cca6cf3325f8471da1d6d63c4443b9fc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44247: connect: Connection refused (error 111)
I20260924 05:04:04.386684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 2478
I20260924 05:04:04.391309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 2094
2026-09-24T05:04:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5548 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:04 dist-test-slave-m01t krb5kdc[2775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:04 dist-test-slave-m01t krb5kdc[2775](info): set up 2 sockets
Sep 24 05:04:04 dist-test-slave-m01t krb5kdc[2775](info): commencing operation
krb5kdc: starting...
W20260924 05:04:05.345178 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:05.447940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:05.524289 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:05.524515 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:05.524547 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:05.525763 2788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:05.525807 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:05.525827 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:05.525843 2788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:05.525861 2788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:05.527413 2788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:05.527673 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:05.527877 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:05.532349 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:05.532356 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.532408 2788 server_base.cc:1034] running on GCE node
W20260924 05:04:05.532356 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.532732 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:05.532974 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:05.534099 2788 hybrid_clock.cc:648] HybridClock initialized: now 1790226245534090 us; error 34 us; skew 500 ppm
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:05.536361 2788 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:05.536783 2788 webserver.cc:460] Webserver started at http://127.0.98.62:39403/ using document root <none> and password file <none>
I20260924 05:04:05.536988 2788 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:05.537032 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:05.537155 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:05.538224 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "3cd21648a1e2304244a46cf4d0e128ef"
server_key_iv: "b39542f10fb1ed8f86ecf9bda704369b"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.538543 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "3cd21648a1e2304244a46cf4d0e128ef"
server_key_iv: "b39542f10fb1ed8f86ecf9bda704369b"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.539736 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:05.540346 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.540484 2788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:05.540565 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "3cd21648a1e2304244a46cf4d0e128ef"
server_key_iv: "b39542f10fb1ed8f86ecf9bda704369b"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.540622 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:05.551142 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:05.552870 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:05.552999 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:05.556582 2788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38783
I20260924 05:04:05.556609 2855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38783 every 8 connection(s)
I20260924 05:04:05.557044 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:05.557610 2856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:05.559715 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4: Bootstrap starting.
I20260924 05:04:05.560218 2856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:05.560448 2856 log.cc:824] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:05.561007 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4: No bootstrap required, opened a new log
I20260924 05:04:05.561586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 2788
I20260924 05:04:05.561664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:05.561827 392 external_mini_cluster.cc:1428] Setting key 16f83c628bc81a686e8e46defacb02c5
I20260924 05:04:05.563199 2856 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38783 } }
I20260924 05:04:05.563359 2856 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:05.563432 2856 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0e3e4ddb0f49cbac81d19f757dfdf4, State: Initialized, Role: FOLLOWER
I20260924 05:04:05.563558 2856 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38783 } }
I20260924 05:04:05.563634 2856 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:05.563656 2856 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:05.563700 2856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:05.564270 2856 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38783 } }
I20260924 05:04:05.564368 2856 leader_election.cc:302] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0e3e4ddb0f49cbac81d19f757dfdf4; no voters:
I20260924 05:04:05.564510 2856 leader_election.cc:288] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:05.564540 2861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:05.564697 2861 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [term 1 LEADER]: Becoming Leader. State: Replica: dd0e3e4ddb0f49cbac81d19f757dfdf4, State: Running, Role: LEADER
I20260924 05:04:05.564725 2856 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:05.564829 2861 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38783 } }
I20260924 05:04:05.565228 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38783 } } }
I20260924 05:04:05.565299 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:05.565965 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0e3e4ddb0f49cbac81d19f757dfdf4. Latest consensus state: current_term: 1 leader_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0e3e4ddb0f49cbac81d19f757dfdf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38783 } } }
I20260924 05:04:05.566082 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:05.566203 2870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:05.566645 2870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:05.568104 2870 catalog_manager.cc:1347] Generated new cluster ID: 74fd6a6a01e74a34ada998ff5000ee9f
I20260924 05:04:05.568157 2870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:05.572861 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:05.562260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.0.98.62:38783)
0924 05:04:05.562433 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:05.562442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:05.562475 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:04:05.563180 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:05.563188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:05.563249 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:05.563392 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:05.564059 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.565389 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.566464 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.566950 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.567052 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:05.569588 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:05.569592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:05.569592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:05.569602 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:05.571383 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:05.571768 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:05.571771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:05.571771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:05.571871 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:05.572049 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:05.572050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:05.572050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:05.572302 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:05.572432 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:05.572542 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 05:04:05.572671 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 05:04:05.584146 2870 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:05.584619 2870 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:05.589226 2870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0e3e4ddb0f49cbac81d19f757dfdf4: Generated new TSK 0
I20260924 05:04:05.589378 2870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:05.618291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38783
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:05.707976 2882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:05.708191 2882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:05.708222 2882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:05.709448 2882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:05.709496 2882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:05.709530 2882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:05.711596 2882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:05.711826 2882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:05.712071 2882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:05.717965 2887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:05.718012 2888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.718281 2882 server_base.cc:1034] running on GCE node
W20260924 05:04:05.718467 2890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.718677 2882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:05.718917 2882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:05.720062 2882 hybrid_clock.cc:648] HybridClock initialized: now 1790226245720046 us; error 29 us; skew 500 ppm
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:05.722361 2882 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:05.722765 2882 webserver.cc:460] Webserver started at http://127.0.98.1:38677/ using document root <none> and password file <none>
I20260924 05:04:05.722980 2882 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:05.723026 2882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:05.723150 2882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:05.724166 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "8abfc540081e4377bfa8364e5f33165d"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "c1a6b8fdce6f64b4bb4925feb312c3a7"
server_key_iv: "b0a9b6caa6f31cd9ff220a05842e3711"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.724490 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8abfc540081e4377bfa8364e5f33165d"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "c1a6b8fdce6f64b4bb4925feb312c3a7"
server_key_iv: "b0a9b6caa6f31cd9ff220a05842e3711"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.725688 2882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:05.726375 2897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.726521 2882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:05.726600 2882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "8abfc540081e4377bfa8364e5f33165d"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "c1a6b8fdce6f64b4bb4925feb312c3a7"
server_key_iv: "b0a9b6caa6f31cd9ff220a05842e3711"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.726660 2882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:05.740094 2882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:05.742017 2882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:05.742159 2882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:05.742381 2882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:05.742707 2882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:05.742743 2882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.742774 2882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:05.742794 2882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.748231 2882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37843
I20260924 05:04:05.748270 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37843 every 8 connection(s)
I20260924 05:04:05.748576 2882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:05.752873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 2882
I20260924 05:04:05.752960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:05.753105 392 external_mini_cluster.cc:1428] Setting key eb8c92d7e4454e9e91630fd49938e98d
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:05.757113 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:05.749232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37697 (local address 127.0.98.62:38783)
0924 05:04:05.749281 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:05.749283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:05.749465 (+ 182us) server_negotiation.cc:407] Connection header received
0924 05:04:05.750370 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:05.750371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:05.750383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:05.750433 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:05.750973 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.751462 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.752356 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.752538 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.752585 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:05.754709 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:05.754710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:05.754711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:05.754717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:05.755939 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:05.756286 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:05.756287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:05.756288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:05.756338 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:05.756542 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:05.756543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:05.756543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:05.756650 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:05.756705 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:05.756948 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 05:04:05.757026 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:04:05.757269 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:05.749209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38783 (local address 127.0.98.1:37697)
0924 05:04:05.749350 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:05.749391 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:05.750293 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:05.750494 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:05.750497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:05.750725 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:05.750907 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:05.750911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.751519 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.751519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:05.752301 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:05.752303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.752594 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.752595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:05.752688 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:05.753196 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:05.753206 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:05.754623 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:05.756012 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:05.756018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:05.756020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:05.756223 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:05.756388 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:05.756388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:05.756388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:05.756489 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:05.756703 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:05.756709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:05.756835 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 05:04:05.756919 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:04:05.757065 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260924 05:04:05.757853 3011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38783
I20260924 05:04:05.757934 3011 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:05.758098 3011 heartbeater.cc:511] Master 127.0.98.62:38783 requested a full tablet report, sending...
I20260924 05:04:05.758610 2820 ts_manager.cc:194] Registered new tserver with Master: 8abfc540081e4377bfa8364e5f33165d (127.0.98.1:37843)
I20260924 05:04:05.759193 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37697
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:05.773730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38783
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:05.852492 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:05.852674 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:05.852694 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:05.853770 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:05.853801 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:05.853824 3016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:05.855218 3016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:05.855391 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:05.855566 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:05.859895 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:05.859907 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:05.859900 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.860136 3016 server_base.cc:1034] running on GCE node
I20260924 05:04:05.860344 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:05.860551 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:05.861685 3016 hybrid_clock.cc:648] HybridClock initialized: now 1790226245861672 us; error 27 us; skew 500 ppm
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:05.863864 3016 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:05.864248 3016 webserver.cc:460] Webserver started at http://127.0.98.2:39657/ using document root <none> and password file <none>
I20260924 05:04:05.864439 3016 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:05.864482 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:05.864593 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:05.865608 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8d347429f4740d998fb9c17da509e1f"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "54922aee7f45804a4f83e418e3f47528"
server_key_iv: "f4f1a23bc66ea827264073b726fe94f9"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.865971 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8d347429f4740d998fb9c17da509e1f"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "54922aee7f45804a4f83e418e3f47528"
server_key_iv: "f4f1a23bc66ea827264073b726fe94f9"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.867213 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:05.867828 3031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.867965 3016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:05.868042 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "c8d347429f4740d998fb9c17da509e1f"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "54922aee7f45804a4f83e418e3f47528"
server_key_iv: "f4f1a23bc66ea827264073b726fe94f9"
server_key_version: "encryptionkey@0"
I20260924 05:04:05.868105 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:05.878945 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:05.880476 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:05.880599 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:05.880781 3016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:05.881049 3016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:05.881083 3016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.881107 3016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:05.881124 3016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:05.886372 3016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45971
I20260924 05:04:05.886423 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45971 every 8 connection(s)
I20260924 05:04:05.886693 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:05.887596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3016
I20260924 05:04:05.887691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:05.887877 392 external_mini_cluster.cc:1428] Setting key 7eb800c4556faa6065a9ce32c9de5f02
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:04:05.895860 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:05.887297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42457 (local address 127.0.98.62:38783)
0924 05:04:05.887345 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:05.887347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:05.887596 (+ 249us) server_negotiation.cc:407] Connection header received
0924 05:04:05.888753 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:05.888754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:05.888763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:05.888799 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:05.889531 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.890231 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.891199 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.891392 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.891447 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:05.893473 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:05.893474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:05.893475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:05.893481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:05.894694 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:05.895031 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:05.895032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:05.895033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:05.895082 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:05.895287 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:05.895288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:05.895288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:05.895391 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:05.895446 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:05.895692 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 05:04:05.895772 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:05.896027 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:05.887289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38783 (local address 127.0.98.2:42457)
0924 05:04:05.887461 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:05.887508 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:05.888659 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:05.888874 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:05.888879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:05.889204 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:05.889460 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:05.889466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.890312 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.890313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:05.891114 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:05.891117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:05.891458 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:05.891459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:05.891590 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:05.892124 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:05.892132 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:05.893389 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:05.894760 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:05.894767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:05.894769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:05.894970 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:05.895133 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:05.895134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:05.895134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:05.895234 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:05.895443 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:05.895449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:05.895580 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 05:04:05.895673 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 05:04:05.895822 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260924 05:04:05.896443 3145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38783
I20260924 05:04:05.896531 3145 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:05.896698 3145 heartbeater.cc:511] Master 127.0.98.62:38783 requested a full tablet report, sending...
I20260924 05:04:05.897050 2820 ts_manager.cc:194] Registered new tserver with Master: c8d347429f4740d998fb9c17da509e1f (127.0.98.2:45971)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 05:04:05.897462 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:05.910287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38783
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:05.987936 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:05.988128 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:05.988149 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:05.989259 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:05.989302 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:05.989331 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:05.990813 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:05.991029 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:05.991236 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:05.995759 3155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:05.995739 3158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.995925 3150 server_base.cc:1034] running on GCE node
W20260924 05:04:05.995935 3156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:05.996173 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:05.996372 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:05.997498 3150 hybrid_clock.cc:648] HybridClock initialized: now 1790226245997490 us; error 24 us; skew 500 ppm
Sep 24 05:04:05 dist-test-slave-m01t krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:05.999778 3150 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:06.000181 3150 webserver.cc:460] Webserver started at http://127.0.98.3:46821/ using document root <none> and password file <none>
I20260924 05:04:06.000365 3150 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:06.000411 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:06.000545 3150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:06.001519 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ec370eefca6424b8a0bfc22ff676c01"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "282e60c13189db9b7a00141b9e7df03d"
server_key_iv: "00350b6530d095660d33441c85d1bbb4"
server_key_version: "encryptionkey@0"
I20260924 05:04:06.001878 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ec370eefca6424b8a0bfc22ff676c01"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "282e60c13189db9b7a00141b9e7df03d"
server_key_iv: "00350b6530d095660d33441c85d1bbb4"
server_key_version: "encryptionkey@0"
I20260924 05:04:06.003050 3150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:04:06.003665 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:06.003827 3150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:06.003904 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "9ec370eefca6424b8a0bfc22ff676c01"
format_stamp: "Formatted at 2026-09-24 05:04:05 on dist-test-slave-m01t"
server_key: "282e60c13189db9b7a00141b9e7df03d"
server_key_iv: "00350b6530d095660d33441c85d1bbb4"
server_key_version: "encryptionkey@0"
I20260924 05:04:06.003962 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:06.011512 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:06.013409 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:06.013532 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:06.013736 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:06.014008 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:06.014040 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:06.014074 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:06.014098 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:06.019228 3150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35595
I20260924 05:04:06.019282 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35595 every 8 connection(s)
I20260924 05:04:06.019549 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:06.024195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3150
I20260924 05:04:06.024294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:06.024473 392 external_mini_cluster.cc:1428] Setting key 02044aeb1ba3f1b1502a3e31b457da17
Sep 24 05:04:06 dist-test-slave-m01t krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:06.028040 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:06.020178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50751 (local address 127.0.98.62:38783)
0924 05:04:06.020230 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:06.020232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:06.020415 (+ 183us) server_negotiation.cc:407] Connection header received
0924 05:04:06.021339 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:06.021340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:06.021352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:06.021417 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:06.022009 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:06.022503 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:06.023433 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:06.023624 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:06.023678 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:06.025916 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:06.025917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:06.025918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:06.025925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:06.027028 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:06.027353 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:06.027354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:06.027354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:06.027400 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:06.027570 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:06.027571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:06.027571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:06.027663 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:06.027708 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:06.027902 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 05:04:06.027974 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:04:06.028224 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:06.020156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38783 (local address 127.0.98.3:50751)
0924 05:04:06.020303 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:06.020342 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:06.021222 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:06.021475 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:06.021479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:06.021747 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:06.021943 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:06.021948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:06.022566 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:06.022567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:06.023354 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:06.023356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:06.023701 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:06.023702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:06.023803 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:06.024291 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:06.024301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:06.025837 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:06.027093 (+ 1256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:06.027098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:06.027099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:06.027299 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:06.027443 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:06.027444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:06.027444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:06.027529 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:06.027704 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:06.027707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:06.027816 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 05:04:06.027889 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 05:04:06.028004 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260924 05:04:06.028623 3279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38783
I20260924 05:04:06.028710 3279 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:06.028878 3279 heartbeater.cc:511] Master 127.0.98.62:38783 requested a full tablet report, sending...
I20260924 05:04:06.029232 2820 ts_manager.cc:194] Registered new tserver with Master: 9ec370eefca6424b8a0bfc22ff676c01 (127.0.98.3:35595)
I20260924 05:04:06.029613 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50751
I20260924 05:04:06.036104 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 05:04:06.045866 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:06.045081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.0.98.62:38783)
0924 05:04:06.045148 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:06.045150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:06.045217 (+ 67us) server_negotiation.cc:407] Connection header received
0924 05:04:06.045263 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:06.045264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:06.045272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:06.045317 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:06.045798 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38656: BlockingRecv error: recv got EOF from 127.0.0.1:38656 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:04:06.046646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 2882
I20260924 05:04:06.050603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3016
I20260924 05:04:06.054401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3150
I20260924 05:04:06.057891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 2788
2026-09-24T05:04:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1665 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:06 dist-test-slave-m01t krb5kdc[3291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:06 dist-test-slave-m01t krb5kdc[3291](info): set up 2 sockets
Sep 24 05:04:06 dist-test-slave-m01t krb5kdc[3291](info): commencing operation
krb5kdc: starting...
W20260924 05:04:06.989195 392 mini_kdc.cc:121] Time spent starting KDC: real 0.913s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:07 dist-test-slave-m01t krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:07.092110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:07.167976 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:07.168164 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:07.168183 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:07.169291 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:07.169327 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:07.169339 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:07.169356 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:07.169385 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:07.170785 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:07.170985 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:07.171176 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:07.175400 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:07.175434 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:07.175436 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:07.175472 3304 server_base.cc:1034] running on GCE node
I20260924 05:04:07.175827 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:07.176086 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:07.177219 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790226247177205 us; error 28 us; skew 500 ppm
Sep 24 05:04:07 dist-test-slave-m01t krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:07.179405 3304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:07.179780 3304 webserver.cc:460] Webserver started at http://127.0.98.62:35995/ using document root <none> and password file <none>
I20260924 05:04:07.179962 3304 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:07.180008 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:07.180135 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:07.181114 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "886ee6f38ec74f40830d47555cd6aea3"
format_stamp: "Formatted at 2026-09-24 05:04:07 on dist-test-slave-m01t"
server_key: "66d5fbf39f967363dd9762da7578103d"
server_key_iv: "fc1f3e906d4ac4061f0e5c58d5b78dce"
server_key_version: "encryptionkey@0"
I20260924 05:04:07.181418 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "886ee6f38ec74f40830d47555cd6aea3"
format_stamp: "Formatted at 2026-09-24 05:04:07 on dist-test-slave-m01t"
server_key: "66d5fbf39f967363dd9762da7578103d"
server_key_iv: "fc1f3e906d4ac4061f0e5c58d5b78dce"
server_key_version: "encryptionkey@0"
I20260924 05:04:07.182590 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 05:04:07.183199 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:07.183334 3304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:07.183409 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "886ee6f38ec74f40830d47555cd6aea3"
format_stamp: "Formatted at 2026-09-24 05:04:07 on dist-test-slave-m01t"
server_key: "66d5fbf39f967363dd9762da7578103d"
server_key_iv: "fc1f3e906d4ac4061f0e5c58d5b78dce"
server_key_version: "encryptionkey@0"
I20260924 05:04:07.183465 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:07.190883 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:07.192414 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:07.192535 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:07.196025 3304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42639
I20260924 05:04:07.196066 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42639 every 8 connection(s)
I20260924 05:04:07.196352 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:07.197080 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:07.199195 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3: Bootstrap starting.
I20260924 05:04:07.199716 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:07.199965 3372 log.cc:824] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:07.200533 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3: No bootstrap required, opened a new log
I20260924 05:04:07.202045 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886ee6f38ec74f40830d47555cd6aea3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 05:04:07.202171 3372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:07.202203 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886ee6f38ec74f40830d47555cd6aea3, State: Initialized, Role: FOLLOWER
I20260924 05:04:07.202335 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886ee6f38ec74f40830d47555cd6aea3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 05:04:07.202390 3372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:07.202411 3372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:07.202453 3372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:07.203145 3372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886ee6f38ec74f40830d47555cd6aea3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 05:04:07.203227 3372 leader_election.cc:302] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886ee6f38ec74f40830d47555cd6aea3; no voters:
I20260924 05:04:07.203366 3372 leader_election.cc:288] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:07.203402 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:07.203516 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [term 1 LEADER]: Becoming Leader. State: Replica: 886ee6f38ec74f40830d47555cd6aea3, State: Running, Role: LEADER
I20260924 05:04:07.203595 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:07.203608 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886ee6f38ec74f40830d47555cd6aea3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } }
I20260924 05:04:07.203975 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886ee6f38ec74f40830d47555cd6aea3. Latest consensus state: current_term: 1 leader_uuid: "886ee6f38ec74f40830d47555cd6aea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886ee6f38ec74f40830d47555cd6aea3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } } }
I20260924 05:04:07.203985 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "886ee6f38ec74f40830d47555cd6aea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886ee6f38ec74f40830d47555cd6aea3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42639 } } }
I20260924 05:04:07.204071 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:07.204077 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:07.204279 3384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:07.204762 3384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:07.205891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3304
I20260924 05:04:07.205983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:07.206149 392 external_mini_cluster.cc:1428] Setting key 4cffd1d9b5bc5949f7bd48f05f523a17
I20260924 05:04:07.206622 3384 catalog_manager.cc:1347] Generated new cluster ID: e4af87119faf46b3b0cf2d5732f55b68
I20260924 05:04:07.206668 3384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:07 dist-test-slave-m01t krb5kdc[3291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:07.216979 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:07.206601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.0.98.62:42639)
0924 05:04:07.206896 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:07.206905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:07.206937 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:04:07.207043 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:07.207049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:07.207102 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:07.207236 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:07.207886 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:07.209307 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:07.210317 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:07.210992 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:07.211107 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:07.213690 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:07.213694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:07.213694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:07.213707 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:07.215483 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:07.215872 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:07.215875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:07.215875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:07.215965 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:07.216139 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:07.216140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:07.216140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:07.216403 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:07.216516 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:07.216636 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 05:04:07.216762 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260924 05:04:07.217808 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:04:07.228628 3384 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:07.229367 3384 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:07.233031 3384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886ee6f38ec74f40830d47555cd6aea3: Generated new TSK 0
I20260924 05:04:07.233217 3384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:07.235741 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:07.232193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.0.98.62:42639)
0924 05:04:07.232233 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:07.232235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:07.232433 (+ 198us) server_negotiation.cc:407] Connection header received
0924 05:04:07.232505 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:07.232506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:07.232517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:07.232582 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:07.233021 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:07.233619 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:07.234398 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:07.234623 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:07.234677 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:07.235321 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:07.235323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:07.235323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:07.235504 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:07.235522 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:07.235528 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:07.235574 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:07.235639 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:04:07.235667 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:07.237886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3304
2026-09-24T05:04:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1178 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:07 dist-test-slave-m01t krb5kdc[3405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:07 dist-test-slave-m01t krb5kdc[3405](info): set up 2 sockets
Sep 24 05:04:07 dist-test-slave-m01t krb5kdc[3405](info): commencing operation
krb5kdc: starting...
W20260924 05:04:08.185247 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:08.289318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:08.366993 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:08.367184 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:08.367213 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:08.368323 3418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:08.368361 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:08.368379 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:08.368395 3418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:08.368410 3418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:08.369825 3418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:08.370044 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:08.370234 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:08.374431 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.374439 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.374445 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:08.374670 3418 server_base.cc:1034] running on GCE node
I20260924 05:04:08.374891 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:08.375160 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:08.376286 3418 hybrid_clock.cc:648] HybridClock initialized: now 1790226248376278 us; error 28 us; skew 500 ppm
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:08.378537 3418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:08.378914 3418 webserver.cc:460] Webserver started at http://127.0.98.62:44445/ using document root <none> and password file <none>
I20260924 05:04:08.379107 3418 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:08.379161 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:08.379276 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:08.380285 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "256ae363951b4e2f8652ec2c8c49b901"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "0e684da9cd58989b8ba618d52e023493"
server_key_iv: "3402699e27c9fba45e94777fbcb60520"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.380607 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "256ae363951b4e2f8652ec2c8c49b901"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "0e684da9cd58989b8ba618d52e023493"
server_key_iv: "3402699e27c9fba45e94777fbcb60520"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.381778 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:08.382393 3433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.382553 3418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:08.382622 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "256ae363951b4e2f8652ec2c8c49b901"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "0e684da9cd58989b8ba618d52e023493"
server_key_iv: "3402699e27c9fba45e94777fbcb60520"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.382680 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:08.390076 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:08.391615 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:08.391731 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:08.395234 3418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34725
I20260924 05:04:08.395263 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34725 every 8 connection(s)
I20260924 05:04:08.395552 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:08.396418 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:08.398608 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901: Bootstrap starting.
I20260924 05:04:08.399138 3486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:08.399389 3486 log.cc:824] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:08.399972 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901: No bootstrap required, opened a new log
I20260924 05:04:08.401477 3486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256ae363951b4e2f8652ec2c8c49b901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34725 } }
I20260924 05:04:08.401608 3486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:08.401646 3486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256ae363951b4e2f8652ec2c8c49b901, State: Initialized, Role: FOLLOWER
I20260924 05:04:08.401808 3486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256ae363951b4e2f8652ec2c8c49b901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34725 } }
I20260924 05:04:08.401883 3486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:08.401916 3486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:08.401959 3486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:08.402673 3486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256ae363951b4e2f8652ec2c8c49b901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34725 } }
I20260924 05:04:08.402776 3486 leader_election.cc:302] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 256ae363951b4e2f8652ec2c8c49b901; no voters:
I20260924 05:04:08.402853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3418
I20260924 05:04:08.402925 3486 leader_election.cc:288] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:08.402949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:08.402971 3489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:08.403101 3489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [term 1 LEADER]: Becoming Leader. State: Replica: 256ae363951b4e2f8652ec2c8c49b901, State: Running, Role: LEADER
I20260924 05:04:08.403127 392 external_mini_cluster.cc:1428] Setting key 24426783e772b2b1a18c32ff04281eb9
I20260924 05:04:08.403165 3486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:08.403180 3489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256ae363951b4e2f8652ec2c8c49b901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34725 } }
I20260924 05:04:08.403527 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 256ae363951b4e2f8652ec2c8c49b901. Latest consensus state: current_term: 1 leader_uuid: "256ae363951b4e2f8652ec2c8c49b901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256ae363951b4e2f8652ec2c8c49b901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34725 } } }
I20260924 05:04:08.403528 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "256ae363951b4e2f8652ec2c8c49b901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256ae363951b4e2f8652ec2c8c49b901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34725 } } }
I20260924 05:04:08.403671 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:08.403683 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:08.403960 3500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:08.404418 3500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:08.406072 3500 catalog_manager.cc:1347] Generated new cluster ID: f05b0fc1aab54a7d9ca9bd5c455db8a4
I20260924 05:04:08.406116 3500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:08.414017 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.403628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.98.62:34725)
0924 05:04:08.403962 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:08.403972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:08.404000 (+ 28us) server_negotiation.cc:407] Connection header received
0924 05:04:08.404105 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:08.404112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:08.404165 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:08.404298 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:08.405056 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.406272 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.407275 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.407874 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.407956 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.410510 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:08.410515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:08.410515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:08.410525 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:08.412352 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.412716 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.412719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.412719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.412836 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.413002 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.413003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.413003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.413305 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:08.413477 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:08.413541 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 05:04:08.413812 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":70,"threads_started":1}
I20260924 05:04:08.419965 3500 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:08.420414 3500 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:08.423305 3500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 256ae363951b4e2f8652ec2c8c49b901: Generated new TSK 0
I20260924 05:04:08.423446 3500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:08.446401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34725
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:08.522624 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:08.522825 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:08.522855 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:08.523999 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:08.524039 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:08.524075 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:08.525506 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:08.525753 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:08.525966 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:08.530556 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.530560 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:08.530604 3512 server_base.cc:1034] running on GCE node
W20260924 05:04:08.530560 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:08.530867 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:08.531111 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:08.532248 3512 hybrid_clock.cc:648] HybridClock initialized: now 1790226248532235 us; error 32 us; skew 500 ppm
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:08.534592 3512 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:08.535015 3512 webserver.cc:460] Webserver started at http://127.0.98.1:45971/ using document root <none> and password file <none>
I20260924 05:04:08.535212 3512 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:08.535259 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:08.535380 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:08.536405 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "a47b53caa60543aeb5b24a9517c11c83"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "59fd8a598ac5ef377e171f568afc0013"
server_key_iv: "286b347bf6dd637048e7804ffd0b8897"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.536737 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a47b53caa60543aeb5b24a9517c11c83"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "59fd8a598ac5ef377e171f568afc0013"
server_key_iv: "286b347bf6dd637048e7804ffd0b8897"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.537942 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:08.538549 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.538690 3512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:08.538775 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "a47b53caa60543aeb5b24a9517c11c83"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "59fd8a598ac5ef377e171f568afc0013"
server_key_iv: "286b347bf6dd637048e7804ffd0b8897"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.538846 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:08.547107 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:08.548648 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:08.548763 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:08.548944 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:08.549209 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:08.549240 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.549259 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:08.549281 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.554443 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41507
I20260924 05:04:08.554479 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41507 every 8 connection(s)
I20260924 05:04:08.554771 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:08.560607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3512
I20260924 05:04:08.560695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:08.560873 392 external_mini_cluster.cc:1428] Setting key 73d7a073a0efc51d543d357ca0d62a39
I20260924 05:04:08.563257 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.555410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40591 (local address 127.0.98.62:34725)
0924 05:04:08.555486 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:08.555488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:08.555633 (+ 145us) server_negotiation.cc:407] Connection header received
0924 05:04:08.556089 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:08.556090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:08.556105 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:08.556152 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:08.556986 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.557476 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.558439 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.558637 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.558680 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.560597 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:08.560598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:08.560598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:08.560605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:08.561993 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.562398 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.562399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.562399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.562445 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.562664 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.562665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.562666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.562799 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:08.562863 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:08.563089 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 05:04:08.563167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:04:08.563328 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.555387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34725 (local address 127.0.98.1:40591)
0924 05:04:08.555525 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:08.555559 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:08.556018 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:08.556209 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:08.556212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:08.556738 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:08.556921 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:08.556926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.557538 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.557538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:08.558371 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:08.558374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.558689 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.558689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:08.558782 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.559231 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:08.559239 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:08.560504 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:08.562062 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:08.562069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:08.562071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:08.562336 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:08.562491 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:08.562491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:08.562492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:08.562607 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:08.562857 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:08.562861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:08.562971 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 05:04:08.563050 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 05:04:08.563166 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260924 05:04:08.563750 3641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34725
I20260924 05:04:08.563844 3641 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:08.564008 3641 heartbeater.cc:511] Master 127.0.98.62:34725 requested a full tablet report, sending...
I20260924 05:04:08.564532 3450 ts_manager.cc:194] Registered new tserver with Master: a47b53caa60543aeb5b24a9517c11c83 (127.0.98.1:41507)
I20260924 05:04:08.565412 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40591
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:08.581934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34725
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:08.659001 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:08.659188 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:08.659209 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:08.660270 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:08.660301 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:08.660326 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:08.661659 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:08.661863 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:08.662030 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:08.666370 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.666365 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.666366 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:08.666558 3646 server_base.cc:1034] running on GCE node
I20260924 05:04:08.666735 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:08.666939 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:08.668063 3646 hybrid_clock.cc:648] HybridClock initialized: now 1790226248668053 us; error 27 us; skew 500 ppm
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:08.670239 3646 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:08.670647 3646 webserver.cc:460] Webserver started at http://127.0.98.2:34853/ using document root <none> and password file <none>
I20260924 05:04:08.670840 3646 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:08.670885 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:08.670998 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:08.671926 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "6aeb90d345264cde9cb65185342fcc5c"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "2e5d97e9b0c2ae15d8e27a70dce287bb"
server_key_iv: "403e5995f69ae0b64028cd2845a59bf1"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.672230 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6aeb90d345264cde9cb65185342fcc5c"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "2e5d97e9b0c2ae15d8e27a70dce287bb"
server_key_iv: "403e5995f69ae0b64028cd2845a59bf1"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.673327 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:08.673950 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.674085 3646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:08.674160 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "6aeb90d345264cde9cb65185342fcc5c"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "2e5d97e9b0c2ae15d8e27a70dce287bb"
server_key_iv: "403e5995f69ae0b64028cd2845a59bf1"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.674218 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:08.703140 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:08.704982 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:08.705117 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:08.705327 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:08.705623 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:08.705664 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.705693 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:08.705739 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.710942 3646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42829
I20260924 05:04:08.710999 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42829 every 8 connection(s)
I20260924 05:04:08.711269 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:08.715780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3646
I20260924 05:04:08.715886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:08.716094 392 external_mini_cluster.cc:1428] Setting key 0477bdc39ae8843ff2c8505af6c8ad91
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:08.719959 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.711882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53303 (local address 127.0.98.62:34725)
0924 05:04:08.711939 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:08.711941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:08.712114 (+ 173us) server_negotiation.cc:407] Connection header received
0924 05:04:08.712609 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:08.712610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:08.712618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:08.712668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:08.713519 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.714046 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.714949 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.715121 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.715169 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.717411 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:08.717412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:08.717413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:08.717419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:08.718666 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.719127 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.719128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.719128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.719180 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.719408 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.719408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.719409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.719498 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:08.719543 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:08.719833 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 05:04:08.719894 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:04:08.720239 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.711846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34725 (local address 127.0.98.2:53303)
0924 05:04:08.711996 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:08.712037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:08.712507 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:08.712730 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:08.712734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:08.713261 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:08.713452 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:08.713457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.714103 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.714104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:08.714891 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:08.714893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.715171 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.715172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:08.715268 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.715670 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:08.715680 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:08.717329 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:08.718765 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:08.718771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:08.718774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:08.719064 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:08.719230 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:08.719230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:08.719231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:08.719357 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:08.719546 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:08.719550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:08.719710 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 05:04:08.719823 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 05:04:08.719985 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 05:04:08.720682 3775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34725
I20260924 05:04:08.720772 3775 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:08.720935 3775 heartbeater.cc:511] Master 127.0.98.62:34725 requested a full tablet report, sending...
I20260924 05:04:08.721302 3450 ts_manager.cc:194] Registered new tserver with Master: 6aeb90d345264cde9cb65185342fcc5c (127.0.98.2:42829)
I20260924 05:04:08.721660 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53303
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:08.738170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34725
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:08.815397 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:08.815588 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:08.815618 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:08.816743 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:08.816781 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:08.816815 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:08.818213 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:08.818431 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:08.818625 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:08.823204 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.823240 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:08.823233 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:08.823401 3780 server_base.cc:1034] running on GCE node
I20260924 05:04:08.823589 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:08.823822 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:08.824949 3780 hybrid_clock.cc:648] HybridClock initialized: now 1790226248824941 us; error 26 us; skew 500 ppm
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:08.827287 3780 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:08.827703 3780 webserver.cc:460] Webserver started at http://127.0.98.3:42765/ using document root <none> and password file <none>
I20260924 05:04:08.827894 3780 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:08.827936 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:08.828056 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:08.829093 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "07bdfbce2c1a4e4fb5aee08649ed6057"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "4a60ac645e54c01aac26e258973d6de1"
server_key_iv: "70062d71948993043f6dcf3e3e70c87d"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.829411 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07bdfbce2c1a4e4fb5aee08649ed6057"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "4a60ac645e54c01aac26e258973d6de1"
server_key_iv: "70062d71948993043f6dcf3e3e70c87d"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.830672 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:08.831322 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.831468 3780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:08.831545 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "07bdfbce2c1a4e4fb5aee08649ed6057"
format_stamp: "Formatted at 2026-09-24 05:04:08 on dist-test-slave-m01t"
server_key: "4a60ac645e54c01aac26e258973d6de1"
server_key_iv: "70062d71948993043f6dcf3e3e70c87d"
server_key_version: "encryptionkey@0"
I20260924 05:04:08.831612 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:08.849087 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:08.850757 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:08.850878 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:08.851087 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:08.851364 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:08.851397 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.851418 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:08.851440 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:08.856693 3780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39787
I20260924 05:04:08.856730 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39787 every 8 connection(s)
I20260924 05:04:08.857012 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:08.862723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3780
I20260924 05:04:08.862826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:08.862994 392 external_mini_cluster.cc:1428] Setting key 604a864e747eea30860cc872bd1747cb
I20260924 05:04:08.864934 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.857638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45905 (local address 127.0.98.62:34725)
0924 05:04:08.857689 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:08.857691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:08.857887 (+ 196us) server_negotiation.cc:407] Connection header received
0924 05:04:08.858355 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:08.858356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:08.858365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:08.858416 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:08.859264 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.859738 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.860653 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.860824 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.860865 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.862678 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:08.862680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:08.862680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:08.862689 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:08.863967 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.864293 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.864294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.864294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.864336 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.864486 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.864487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.864487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.864576 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:08.864617 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:08.864822 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 05:04:08.864874 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:04:08.865103 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.857616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34725 (local address 127.0.98.3:45905)
0924 05:04:08.857770 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:08.857810 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:08.858281 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:08.858469 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:08.858473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:08.858998 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:08.859194 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:08.859200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.859802 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.859802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:08.860593 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:08.860594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.860874 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.860875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:08.860963 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.861356 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:08.861363 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:08.862589 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:08.864029 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:08.864034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:08.864036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:08.864239 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:08.864370 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:08.864370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:08.864370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:08.864449 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:08.864619 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:08.864622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:08.864726 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 05:04:08.864803 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:04:08.864911 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260924 05:04:08.865490 3909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34725
I20260924 05:04:08.865572 3909 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:08.865775 3909 heartbeater.cc:511] Master 127.0.98.62:34725 requested a full tablet report, sending...
I20260924 05:04:08.866108 3450 ts_manager.cc:194] Registered new tserver with Master: 07bdfbce2c1a4e4fb5aee08649ed6057 (127.0.98.3:39787)
I20260924 05:04:08.866458 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45905
I20260924 05:04:08.874413 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226248, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:08.894459 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:08.888212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.62:34725)
0924 05:04:08.888259 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:08.888261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:08.888363 (+ 102us) server_negotiation.cc:407] Connection header received
0924 05:04:08.888428 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:08.888429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:08.888440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:08.888484 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:08.889000 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.889479 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.890236 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:08.890420 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:08.890474 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:08.892437 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:08.892439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:08.892439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:08.892447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:08.893453 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.893791 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.893792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.893793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.893849 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:08.894045 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:08.894045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:08.894046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:08.894191 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:08.894260 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:08.894338 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 05:04:08.894396 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:04:08.894682 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43886
I20260924 05:04:08.895149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3512
I20260924 05:04:08.898813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3646
I20260924 05:04:08.902408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3780
I20260924 05:04:08.905833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3418
2026-09-24T05:04:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1669 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3921](info): set up 2 sockets
Sep 24 05:04:08 dist-test-slave-m01t krb5kdc[3921](info): commencing operation
krb5kdc: starting...
W20260924 05:04:09.849201 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:09 dist-test-slave-m01t krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:09.953640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:10.030334 3934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:10.030556 3934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:10.030580 3934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:10.031733 3934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:10.031769 3934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:10.031782 3934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:10.031793 3934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:10.031807 3934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:10.033231 3934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:10.033434 3934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:10.033620 3934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:04:10.038188 3934 server_base.cc:1034] running on GCE node
W20260924 05:04:10.038153 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:10.038136 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:10.038149 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:10.038635 3934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:10.038913 3934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:10.040042 3934 hybrid_clock.cc:648] HybridClock initialized: now 1790226250040029 us; error 31 us; skew 500 ppm
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:10.042346 3934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:10.042791 3934 webserver.cc:460] Webserver started at http://127.0.98.62:36585/ using document root <none> and password file <none>
I20260924 05:04:10.042999 3934 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:10.043046 3934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:10.043162 3934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:10.044158 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "147c51cd35bd483b9821f37561faf8b8"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "a68d13a9261b70b148928aaa6404e6ea"
server_key_iv: "8daaf41b8fdcb16ee83ac76655a1e122"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.044462 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "147c51cd35bd483b9821f37561faf8b8"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "a68d13a9261b70b148928aaa6404e6ea"
server_key_iv: "8daaf41b8fdcb16ee83ac76655a1e122"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.045995 3934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:10.046633 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.046816 3934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:10.046895 3934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "147c51cd35bd483b9821f37561faf8b8"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "a68d13a9261b70b148928aaa6404e6ea"
server_key_iv: "8daaf41b8fdcb16ee83ac76655a1e122"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.046954 3934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:10.054863 3934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:10.056382 3934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:10.056500 3934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:10.059986 3934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34157
I20260924 05:04:10.060021 4001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34157 every 8 connection(s)
I20260924 05:04:10.060302 3934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:10.061060 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:10.063139 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8: Bootstrap starting.
I20260924 05:04:10.063617 4002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:10.063840 4002 log.cc:824] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:10.064383 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8: No bootstrap required, opened a new log
I20260924 05:04:10.065891 4002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c51cd35bd483b9821f37561faf8b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 05:04:10.066017 4002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:10.066037 4002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147c51cd35bd483b9821f37561faf8b8, State: Initialized, Role: FOLLOWER
I20260924 05:04:10.066146 4002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c51cd35bd483b9821f37561faf8b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 05:04:10.066216 4002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:10.066243 4002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:10.066274 4002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:10.066715 4002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c51cd35bd483b9821f37561faf8b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 05:04:10.066792 4002 leader_election.cc:302] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147c51cd35bd483b9821f37561faf8b8; no voters:
I20260924 05:04:10.066903 4002 leader_election.cc:288] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:10.066958 4005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:10.067127 4002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:10.067130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 3934
I20260924 05:04:10.067260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:10.067278 4005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 147c51cd35bd483b9821f37561faf8b8, State: Running, Role: LEADER
I20260924 05:04:10.067363 4005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c51cd35bd483b9821f37561faf8b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } }
I20260924 05:04:10.067436 392 external_mini_cluster.cc:1428] Setting key 8ca739830c315a9b62b8a0804e2eccc0
I20260924 05:04:10.067673 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147c51cd35bd483b9821f37561faf8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c51cd35bd483b9821f37561faf8b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } } }
I20260924 05:04:10.067745 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:10.067893 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147c51cd35bd483b9821f37561faf8b8. Latest consensus state: current_term: 1 leader_uuid: "147c51cd35bd483b9821f37561faf8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c51cd35bd483b9821f37561faf8b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34157 } } }
I20260924 05:04:10.067940 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:10.068495 4015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:10.068991 4015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:10.070418 4015 catalog_manager.cc:1347] Generated new cluster ID: 7fe6e3d8fdd74c70b1f8d10b99f9167a
I20260924 05:04:10.070467 4015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:10.078183 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.067997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.98.62:34157)
0924 05:04:10.068202 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:10.068212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:10.068315 (+ 103us) server_negotiation.cc:407] Connection header received
0924 05:04:10.068717 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:10.068724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:10.068766 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:10.068876 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:10.069479 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.070686 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.071373 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.071774 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.071873 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.073986 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:10.073990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:10.073991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:10.074000 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:10.076428 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.076828 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.076831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.076832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.076932 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.077134 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.077135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.077136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.077423 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:10.077558 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:10.077833 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 05:04:10.077967 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260924 05:04:10.078662 4015 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:10.079246 4015 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:10.082974 4015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147c51cd35bd483b9821f37561faf8b8: Generated new TSK 0
I20260924 05:04:10.083112 4015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:10.108999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34157
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:10.188542 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:10.188755 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:10.188786 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:10.189934 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:10.189973 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:10.190007 4028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:10.191423 4028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:10.191663 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:10.191867 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:10.196367 4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:10.196384 4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:10.196375 4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:10.196586 4028 server_base.cc:1034] running on GCE node
I20260924 05:04:10.196813 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:10.197057 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:10.198213 4028 hybrid_clock.cc:648] HybridClock initialized: now 1790226250198199 us; error 30 us; skew 500 ppm
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:10.200440 4028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:10.200803 4028 webserver.cc:460] Webserver started at http://127.0.98.1:41839/ using document root <none> and password file <none>
I20260924 05:04:10.200969 4028 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:10.201007 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:10.201097 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:10.202102 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "48531fa0ac0b407e835f342908e9a314"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "7f29f81e6a236b22b9f11735a64f349c"
server_key_iv: "6c45edd8bafaaf2f152758442fce68e4"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.202389 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48531fa0ac0b407e835f342908e9a314"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "7f29f81e6a236b22b9f11735a64f349c"
server_key_iv: "6c45edd8bafaaf2f152758442fce68e4"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.203493 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:10.204095 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.204238 4028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:10.204313 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "48531fa0ac0b407e835f342908e9a314"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "7f29f81e6a236b22b9f11735a64f349c"
server_key_iv: "6c45edd8bafaaf2f152758442fce68e4"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.204370 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:10.216678 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:10.218317 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:10.218444 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:10.218657 4028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:10.218927 4028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:10.218961 4028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.218988 4028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:10.219012 4028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.224514 4028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34097
I20260924 05:04:10.224576 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34097 every 8 connection(s)
I20260924 05:04:10.224877 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:10.233151 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.225570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53639 (local address 127.0.98.62:34157)
0924 05:04:10.225614 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:10.225616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:10.225825 (+ 209us) server_negotiation.cc:407] Connection header received
0924 05:04:10.226742 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:10.226743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:10.226752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:10.226799 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:10.227386 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.227885 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.228807 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.228994 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.229036 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.231036 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:10.231037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:10.231038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:10.231047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:10.232144 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.232469 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.232470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.232470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.232511 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.232679 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.232680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.232680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.232777 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:10.232821 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:10.233022 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 05:04:10.233084 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:10.233342 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.225535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34157 (local address 127.0.98.1:53639)
0924 05:04:10.225684 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:10.225752 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:10.226656 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:10.226864 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:10.226868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:10.227121 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:10.227323 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:10.227327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.227946 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.227947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:10.228745 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:10.228747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.229046 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.229047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:10.229149 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.229636 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:10.229644 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:10.230953 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:10.232206 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:10.232211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:10.232213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:10.232417 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:10.232553 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:10.232553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:10.232554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:10.232638 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:10.232816 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:10.232819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:10.232933 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 05:04:10.233007 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:04:10.233122 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260924 05:04:10.233644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4028
I20260924 05:04:10.233767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:10.233887 4157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34157
I20260924 05:04:10.233960 392 external_mini_cluster.cc:1428] Setting key 5503d2344009410893db3d1f8c651eb6
I20260924 05:04:10.233987 4157 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:10.234176 4157 heartbeater.cc:511] Master 127.0.98.62:34157 requested a full tablet report, sending...
I20260924 05:04:10.234719 3966 ts_manager.cc:194] Registered new tserver with Master: 48531fa0ac0b407e835f342908e9a314 (127.0.98.1:34097)
I20260924 05:04:10.235302 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53639
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:10.253566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34157
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:10.329918 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:10.330132 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:10.330169 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:10.331333 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:10.331377 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:10.331409 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:10.332875 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:10.333115 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:10.333338 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:10.338384 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:10.338474 4162 server_base.cc:1034] running on GCE node
W20260924 05:04:10.338522 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:10.338408 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:10.338771 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:10.338987 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:10.340118 4162 hybrid_clock.cc:648] HybridClock initialized: now 1790226250340110 us; error 33 us; skew 500 ppm
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:10.342587 4162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:10.343029 4162 webserver.cc:460] Webserver started at http://127.0.98.2:34939/ using document root <none> and password file <none>
I20260924 05:04:10.343232 4162 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:10.343292 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:10.343417 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:10.344754 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "69ce22fc9057424e89245a38109e8e87"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "02bbdeb536850cebf942004a529c1621"
server_key_iv: "34ed706a000532260dda98fbb2bc5b9c"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.345163 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "69ce22fc9057424e89245a38109e8e87"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "02bbdeb536850cebf942004a529c1621"
server_key_iv: "34ed706a000532260dda98fbb2bc5b9c"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.346385 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:04:10.347007 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.347137 4162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:10.347213 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "69ce22fc9057424e89245a38109e8e87"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "02bbdeb536850cebf942004a529c1621"
server_key_iv: "34ed706a000532260dda98fbb2bc5b9c"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.347271 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:10.358461 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:10.359949 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:10.360071 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:10.360244 4162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:10.360499 4162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:10.360531 4162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.360556 4162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:10.360574 4162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.365656 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43477
I20260924 05:04:10.365748 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43477 every 8 connection(s)
I20260924 05:04:10.365969 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:10.367264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4162
I20260924 05:04:10.367352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:10.367498 392 external_mini_cluster.cc:1428] Setting key 2891f49f1caf26c1d3682a6078b63c0b
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:04:10.375136 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.366581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53933 (local address 127.0.98.62:34157)
0924 05:04:10.366642 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:10.366644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:10.366811 (+ 167us) server_negotiation.cc:407] Connection header received
0924 05:04:10.367758 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:10.367760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:10.367769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:10.367847 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:10.368628 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.369156 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.370304 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.370491 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.370541 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.372685 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:10.372686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:10.372686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:10.372693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:10.373881 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.374289 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.374290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.374290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.374335 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.374567 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.374568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.374568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.374662 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:10.374705 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:10.374999 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 05:04:10.375056 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:04:10.375355 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.366571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34157 (local address 127.0.98.2:53933)
0924 05:04:10.366712 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:10.366743 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:10.367657 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:10.367966 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:10.367970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:10.368300 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:10.368549 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:10.368556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.369233 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.369234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:10.370224 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:10.370227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.370559 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.370560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:10.370697 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.371118 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:10.371127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:10.372601 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:10.373947 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:10.373953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:10.373956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:10.374223 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:10.374386 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:10.374386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:10.374387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:10.374510 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:10.374714 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:10.374719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:10.374874 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 05:04:10.374981 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 05:04:10.375140 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260924 05:04:10.375754 4291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34157
I20260924 05:04:10.375831 4291 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:10.376006 4291 heartbeater.cc:511] Master 127.0.98.62:34157 requested a full tablet report, sending...
I20260924 05:04:10.376334 3966 ts_manager.cc:194] Registered new tserver with Master: 69ce22fc9057424e89245a38109e8e87 (127.0.98.2:43477)
I20260924 05:04:10.376694 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:10.387787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34157
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:10.466059 4296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:10.466251 4296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:10.466272 4296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:10.467497 4296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:10.467536 4296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:10.467566 4296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:10.469044 4296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:10.469254 4296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:10.469484 4296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:10.473938 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:10.473945 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:10.474085 4296 server_base.cc:1034] running on GCE node
W20260924 05:04:10.473938 4304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:10.474313 4296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:10.474511 4296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:10.475643 4296 hybrid_clock.cc:648] HybridClock initialized: now 1790226250475622 us; error 32 us; skew 500 ppm
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:10.477854 4296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:10.478215 4296 webserver.cc:460] Webserver started at http://127.0.98.3:35705/ using document root <none> and password file <none>
I20260924 05:04:10.478370 4296 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:10.478407 4296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:10.478487 4296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:10.479393 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2a7f14f987347b8800f594fa2c0490d"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "3a657e2a0c05f762a3c92805ab8534e7"
server_key_iv: "0d7c4ef672ea8240dbf65dc7f39011ec"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.479671 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2a7f14f987347b8800f594fa2c0490d"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "3a657e2a0c05f762a3c92805ab8534e7"
server_key_iv: "0d7c4ef672ea8240dbf65dc7f39011ec"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.480769 4296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:10.481362 4311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.481498 4296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.481586 4296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "f2a7f14f987347b8800f594fa2c0490d"
format_stamp: "Formatted at 2026-09-24 05:04:10 on dist-test-slave-m01t"
server_key: "3a657e2a0c05f762a3c92805ab8534e7"
server_key_iv: "0d7c4ef672ea8240dbf65dc7f39011ec"
server_key_version: "encryptionkey@0"
I20260924 05:04:10.481640 4296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:10.493861 4296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:10.495445 4296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:10.495554 4296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:10.495719 4296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:10.495952 4296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:10.495980 4296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.495999 4296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:10.496016 4296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:10.501389 4296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40857
I20260924 05:04:10.501438 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40857 every 8 connection(s)
I20260924 05:04:10.501746 4296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:10.509505 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.502334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47569 (local address 127.0.98.62:34157)
0924 05:04:10.502404 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:10.502407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:10.502555 (+ 148us) server_negotiation.cc:407] Connection header received
0924 05:04:10.503404 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:10.503405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:10.503415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:10.503459 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:10.503982 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.504465 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.505356 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.505526 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.505569 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.507484 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:10.507485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:10.507486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:10.507492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:10.508550 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.508861 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.508861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.508862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.508903 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.509066 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.509066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.509066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.509156 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:10.509199 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:10.509387 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 05:04:10.509442 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:04:10.509668 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.502316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34157 (local address 127.0.98.3:47569)
0924 05:04:10.502452 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:10.502484 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:10.503323 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:10.503517 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:10.503520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:10.503747 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:10.503925 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:10.503930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.504522 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.504523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:10.505300 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:10.505303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.505576 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.505577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:10.505665 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.506125 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:10.506142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:10.507407 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:10.508610 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:10.508615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:10.508617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:10.508809 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:10.508944 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:10.508944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:10.508944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:10.509025 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:10.509196 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:10.509198 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:10.509299 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 05:04:10.509370 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:04:10.509475 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260924 05:04:10.510166 4425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34157
I20260924 05:04:10.510251 4425 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:10.510406 4425 heartbeater.cc:511] Master 127.0.98.62:34157 requested a full tablet report, sending...
I20260924 05:04:10.510718 3966 ts_manager.cc:194] Registered new tserver with Master: f2a7f14f987347b8800f594fa2c0490d (127.0.98.3:40857)
I20260924 05:04:10.511050 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47569
I20260924 05:04:10.511559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4296
I20260924 05:04:10.511641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:10.511811 392 external_mini_cluster.cc:1428] Setting key 104f5400262fdd4889e3022f81af1ecd
I20260924 05:04:10.512445 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:04:10.532814 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.526019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.0.98.1:34097)
0924 05:04:10.526160 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:10.526165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:10.526176 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:10.526202 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:10.526204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:10.526217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:10.526277 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:10.526792 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.527502 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.528177 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.528478 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.528526 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.530516 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:10.530526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:10.530526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:10.530533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:10.531976 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.532233 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.532235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.532236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.532285 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.532399 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.532400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.532400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.532523 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:10.532621 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:10.532673 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 05:04:10.532733 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:10.539857 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:10.534024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.0.98.62:34157)
0924 05:04:10.534068 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:10.534070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:10.534112 (+ 42us) server_negotiation.cc:407] Connection header received
0924 05:04:10.534172 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:10.534173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:10.534182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:10.534217 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:10.534616 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.535099 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.535718 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:10.535887 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:10.535929 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:10.537868 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:10.537870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:10.537870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:10.537877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:10.539052 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.539305 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.539306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.539306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.539351 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:10.539466 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:10.539467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:10.539467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:10.539594 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:10.539667 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:10.539728 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 05:04:10.539784 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:04:10.540091 3966 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47114
I20260924 05:04:10.540416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4028
I20260924 05:04:10.544687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4162
I20260924 05:04:10.548403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4296
I20260924 05:04:10.551776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 3934
2026-09-24T05:04:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1645 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[4437](info): set up 2 sockets
Sep 24 05:04:10 dist-test-slave-m01t krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260924 05:04:11.497304 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:11 dist-test-slave-m01t krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:11.600294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33849
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:11.676131 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:11.676332 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:11.676352 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:11.677430 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:11.677460 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:11.677472 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:11.677484 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:11.677493 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:11.678951 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:11.679128 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:11.679301 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:11.683652 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:11.683696 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:11.683710 4450 server_base.cc:1034] running on GCE node
W20260924 05:04:11.683657 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:11.684022 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:11.684293 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:11.685423 4450 hybrid_clock.cc:648] HybridClock initialized: now 1790226251685412 us; error 26 us; skew 500 ppm
Sep 24 05:04:11 dist-test-slave-m01t krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:11.692462 4450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:11.692804 4450 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:11.692857 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:11.692976 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:11.694022 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "e432cc4e519d4d22b9a17948c0b80af5"
format_stamp: "Formatted at 2026-09-24 05:04:11 on dist-test-slave-m01t"
server_key: "8193857abfe2d47de6db2203fb1732ee"
server_key_iv: "a3047d6e92e895037e90818909173ae3"
server_key_version: "encryptionkey@0"
I20260924 05:04:11.694335 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "e432cc4e519d4d22b9a17948c0b80af5"
format_stamp: "Formatted at 2026-09-24 05:04:11 on dist-test-slave-m01t"
server_key: "8193857abfe2d47de6db2203fb1732ee"
server_key_iv: "a3047d6e92e895037e90818909173ae3"
server_key_version: "encryptionkey@0"
I20260924 05:04:11.695470 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:11.696069 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:11.696197 4450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:11.696271 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e432cc4e519d4d22b9a17948c0b80af5"
format_stamp: "Formatted at 2026-09-24 05:04:11 on dist-test-slave-m01t"
server_key: "8193857abfe2d47de6db2203fb1732ee"
server_key_iv: "a3047d6e92e895037e90818909173ae3"
server_key_version: "encryptionkey@0"
I20260924 05:04:11.696327 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:11.706848 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:11.708360 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:11.708477 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:11.712548 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33849
I20260924 05:04:11.712567 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33849 every 8 connection(s)
I20260924 05:04:11.712848 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:11.713769 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:11.713821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4450
I20260924 05:04:11.713920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:11.714104 392 external_mini_cluster.cc:1428] Setting key abb9af5095c8fe57ccf10829d13d18c4
I20260924 05:04:11.716295 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: Bootstrap starting.
I20260924 05:04:11.716989 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:11.717275 4517 log.cc:824] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:11.718560 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: No bootstrap required, opened a new log
I20260924 05:04:11.720763 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432cc4e519d4d22b9a17948c0b80af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260924 05:04:11.720906 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:11.720937 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e432cc4e519d4d22b9a17948c0b80af5, State: Initialized, Role: FOLLOWER
I20260924 05:04:11.721062 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432cc4e519d4d22b9a17948c0b80af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260924 05:04:11.721138 4517 raft_consensus.cc:388] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:11.721168 4517 raft_consensus.cc:482] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:11.721213 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 05:04:11 dist-test-slave-m01t krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:11.721911 4517 raft_consensus.cc:504] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432cc4e519d4d22b9a17948c0b80af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260924 05:04:11.722018 4517 leader_election.cc:302] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e432cc4e519d4d22b9a17948c0b80af5; no voters:
I20260924 05:04:11.722169 4517 leader_election.cc:288] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:11.722198 4522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:11.722312 4522 raft_consensus.cc:686] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [term 1 LEADER]: Becoming Leader. State: Replica: e432cc4e519d4d22b9a17948c0b80af5, State: Running, Role: LEADER
I20260924 05:04:11.722399 4522 consensus_queue.cc:237] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432cc4e519d4d22b9a17948c0b80af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260924 05:04:11.722430 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:11.722730 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e432cc4e519d4d22b9a17948c0b80af5. Latest consensus state: current_term: 1 leader_uuid: "e432cc4e519d4d22b9a17948c0b80af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432cc4e519d4d22b9a17948c0b80af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } } }
I20260924 05:04:11.722744 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e432cc4e519d4d22b9a17948c0b80af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432cc4e519d4d22b9a17948c0b80af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } } }
I20260924 05:04:11.722812 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:11.722856 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5 [sys.catalog]: This master's current role is: LEADER
W20260924 05:04:11.724212 4538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:04:11.724267 4538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:04:11.724316 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:11.724473 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:11.725996 4527 catalog_manager.cc:1347] Generated new cluster ID: 9c531080615743d6b80b65a7c882a360
I20260924 05:04:11.726060 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:11.726706 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:11.714714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.98.62:33849)
0924 05:04:11.714873 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:11.714882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:11.714911 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:04:11.715442 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:11.715451 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:11.715511 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:11.715696 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:11.716401 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:11.718156 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:11.719212 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:11.719816 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:11.719926 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:11.722348 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:11.722352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:11.722352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:11.722364 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:11.725077 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:11.725455 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:11.725458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:11.725459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:11.725561 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:11.725794 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:11.725795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:11.725796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:11.726069 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:11.726211 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:11.726411 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 05:04:11.726543 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260924 05:04:11.762413 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:11.763175 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:11.766317 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: Generated new TSK 0
I20260924 05:04:11.766492 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:11.789376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:11.867094 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:11.867288 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:11.867311 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:11.868460 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:11.868500 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:11.868530 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:11.870023 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:11.870232 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:11.870428 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:11.874935 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:11.874948 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:11.875005 4543 server_base.cc:1034] running on GCE node
W20260924 05:04:11.874940 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:11.875342 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:11.875560 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:11.876694 4543 hybrid_clock.cc:648] HybridClock initialized: now 1790226251876674 us; error 36 us; skew 500 ppm
Sep 24 05:04:11 dist-test-slave-m01t krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:11.879091 4543 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:11.879407 4543 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:11.879456 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:11.879545 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:11.880545 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "fbee93eb09c845f9b3444262364a19a3"
format_stamp: "Formatted at 2026-09-24 05:04:11 on dist-test-slave-m01t"
server_key: "04f0da4a6cce8452dd3de2f41d501756"
server_key_iv: "21941c36585fcaaeef7a006f6c534bd0"
server_key_version: "encryptionkey@0"
I20260924 05:04:11.880844 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fbee93eb09c845f9b3444262364a19a3"
format_stamp: "Formatted at 2026-09-24 05:04:11 on dist-test-slave-m01t"
server_key: "04f0da4a6cce8452dd3de2f41d501756"
server_key_iv: "21941c36585fcaaeef7a006f6c534bd0"
server_key_version: "encryptionkey@0"
I20260924 05:04:11.881974 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:11.882597 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:11.882740 4543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:11.882805 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "fbee93eb09c845f9b3444262364a19a3"
format_stamp: "Formatted at 2026-09-24 05:04:11 on dist-test-slave-m01t"
server_key: "04f0da4a6cce8452dd3de2f41d501756"
server_key_iv: "21941c36585fcaaeef7a006f6c534bd0"
server_key_version: "encryptionkey@0"
I20260924 05:04:11.882872 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:11.891029 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:11.893083 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:11.893213 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:11.893411 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:11.893692 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:11.893757 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:11.893795 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:11.893819 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:11.899197 4543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34311
I20260924 05:04:11.899241 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34311 every 8 connection(s)
I20260924 05:04:11.899475 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:11.903589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4543
I20260924 05:04:11.903680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:11.903841 392 external_mini_cluster.cc:1428] Setting key 2edaf06046e4ae78f717c8de377a3d7c
Sep 24 05:04:11 dist-test-slave-m01t krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:11.908203 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:11.900058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46379 (local address 127.0.98.62:33849)
0924 05:04:11.900099 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:11.900102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:11.900275 (+ 173us) server_negotiation.cc:407] Connection header received
0924 05:04:11.900777 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:11.900778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:11.900790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:11.900848 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:11.901749 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:11.902242 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:11.903198 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:11.903415 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:11.903484 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:11.905625 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:11.905626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:11.905627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:11.905634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:11.906918 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:11.907324 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:11.907325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:11.907325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:11.907367 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:11.907626 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:11.907627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:11.907627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:11.907726 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:11.907771 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:11.908069 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 05:04:11.908140 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:11.908463 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:11.900035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33849 (local address 127.0.98.1:46379)
0924 05:04:11.900161 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:11.900198 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:11.900683 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:11.900928 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:11.900932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:11.901180 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:11.901646 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:11.901650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:11.902322 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:11.902323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:11.903120 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:11.903122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:11.903512 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:11.903513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:11.903621 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:11.904086 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:11.904095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:11.905527 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:11.906991 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:11.906997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:11.906999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:11.907258 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:11.907425 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:11.907426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:11.907426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:11.907561 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:11.907781 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:11.907786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:11.907945 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 05:04:11.908058 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 05:04:11.908213 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260924 05:04:11.909026 4671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33849
I20260924 05:04:11.909124 4671 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:11.909281 4671 heartbeater.cc:511] Master 127.0.98.62:33849 requested a full tablet report, sending...
I20260924 05:04:11.909802 4479 ts_manager.cc:194] Registered new tserver with Master: fbee93eb09c845f9b3444262364a19a3 (127.0.98.1:34311)
I20260924 05:04:11.910380 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46379
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:11.925796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:12.003731 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:12.003921 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:12.003942 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:12.005049 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:12.005093 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:12.005122 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:12.006582 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:12.006788 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:12.006985 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:12.011374 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.011379 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.011452 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.011601 4676 server_base.cc:1034] running on GCE node
I20260924 05:04:12.011795 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:12.011984 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:12.013113 4676 hybrid_clock.cc:648] HybridClock initialized: now 1790226252013104 us; error 26 us; skew 500 ppm
Sep 24 05:04:12 dist-test-slave-m01t krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:12.015405 4676 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:12.015691 4676 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:12.015734 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:12.015846 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:12.016793 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e38c37f2cdf46c6915f199f52e927fb"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "3be930be1a4323e58cab143e8dd4f6e2"
server_key_iv: "bddd3acd8dc23b82b4ec3b53fa09fa9c"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.017091 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e38c37f2cdf46c6915f199f52e927fb"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "3be930be1a4323e58cab143e8dd4f6e2"
server_key_iv: "bddd3acd8dc23b82b4ec3b53fa09fa9c"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.018232 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:12.018822 4690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.018962 4676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:12.019039 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "9e38c37f2cdf46c6915f199f52e927fb"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "3be930be1a4323e58cab143e8dd4f6e2"
server_key_iv: "bddd3acd8dc23b82b4ec3b53fa09fa9c"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.019093 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:12.038970 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:12.040764 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:12.040876 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:12.041060 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:12.041316 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:12.041344 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.041364 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:12.041378 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.046502 4676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45105
I20260924 05:04:12.046568 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45105 every 8 connection(s)
I20260924 05:04:12.046793 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:12.049736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4676
I20260924 05:04:12.049839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:12.050017 392 external_mini_cluster.cc:1428] Setting key 11c31a94306909cfa6813e14a7fedcc8
Sep 24 05:04:12 dist-test-slave-m01t krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:12.055686 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.047430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53967 (local address 127.0.98.62:33849)
0924 05:04:12.047492 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.047494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.047682 (+ 188us) server_negotiation.cc:407] Connection header received
0924 05:04:12.048120 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.048122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.048131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.048183 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.049019 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.049501 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.050805 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.050989 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.051039 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.053239 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.053240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.053240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:12.053247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.054510 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:12.054848 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:12.054849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:12.054849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:12.054899 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:12.055105 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:12.055106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:12.055106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:12.055210 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.055263 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.055504 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.055589 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:04:12.055820 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.047401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33849 (local address 127.0.98.2:53967)
0924 05:04:12.047552 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.047593 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.048050 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.048236 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.048239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.048481 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.048956 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.048960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.049580 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.049581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.050734 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.050737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.051055 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.051057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.051189 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.051623 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:12.051640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.053156 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.054580 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:12.054586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:12.054588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:12.054787 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:12.054950 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:12.054951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:12.054951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:12.055050 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:12.055261 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.055268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.055395 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.055480 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.055617 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260924 05:04:12.056169 4804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33849
I20260924 05:04:12.056253 4804 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:12.056388 4804 heartbeater.cc:511] Master 127.0.98.62:33849 requested a full tablet report, sending...
I20260924 05:04:12.056707 4479 ts_manager.cc:194] Registered new tserver with Master: 9e38c37f2cdf46c6915f199f52e927fb (127.0.98.2:45105)
I20260924 05:04:12.057060 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53967
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:12.070315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:12.148196 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:12.148391 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:12.148420 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:12.149533 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:12.149572 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:12.149607 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:12.151077 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:12.151301 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:12.151494 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:12.155889 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.155905 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.155936 4809 server_base.cc:1034] running on GCE node
W20260924 05:04:12.155889 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.156308 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:12.156490 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:12.157611 4809 hybrid_clock.cc:648] HybridClock initialized: now 1790226252157603 us; error 25 us; skew 500 ppm
Sep 24 05:04:12 dist-test-slave-m01t krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:12.159790 4809 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:12.160070 4809 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:12.160111 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:12.160215 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:12.161162 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "0597242acde7490e86e300678938cfae"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "0541394d803076844393857eb178b806"
server_key_iv: "c51dca71b2727a300e69eb8ba0177c4b"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.161458 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0597242acde7490e86e300678938cfae"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "0541394d803076844393857eb178b806"
server_key_iv: "c51dca71b2727a300e69eb8ba0177c4b"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.162606 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:04:12.163213 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.163350 4809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:12.163421 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "0597242acde7490e86e300678938cfae"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "0541394d803076844393857eb178b806"
server_key_iv: "c51dca71b2727a300e69eb8ba0177c4b"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.163480 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:12.177171 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:12.178781 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:12.178900 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:12.179087 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:12.179366 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:12.179399 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.179425 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:12.179441 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.184542 4809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38269
I20260924 05:04:12.184599 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38269 every 8 connection(s)
I20260924 05:04:12.184823 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:04:12 dist-test-slave-m01t krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:12.192844 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.185460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60589 (local address 127.0.98.62:33849)
0924 05:04:12.185538 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.185540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.185696 (+ 156us) server_negotiation.cc:407] Connection header received
0924 05:04:12.186198 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.186199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.186208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.186257 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.187185 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.187666 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.188625 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.188805 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.188847 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.190784 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.190785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.190785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:12.190791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.191841 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:12.192174 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:12.192174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:12.192175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:12.192217 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:12.192386 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:12.192386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:12.192387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:12.192478 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.192520 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.192726 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.192779 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 05:04:12.193029 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.185431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33849 (local address 127.0.98.3:60589)
0924 05:04:12.185583 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.185622 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.186118 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.186318 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.186322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.186596 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.187114 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.187120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.187730 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.187730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.188565 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.188568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.188858 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.188858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.188949 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.189402 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:12.189409 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.190712 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.191901 (+ 1189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:12.191906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:12.191908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:12.192121 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:12.192260 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:12.192260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:12.192260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:12.192346 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:12.192518 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.192521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.192632 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.192709 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.192822 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260924 05:04:12.193462 4937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33849
I20260924 05:04:12.193552 4937 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:12.193667 4937 heartbeater.cc:511] Master 127.0.98.62:33849 requested a full tablet report, sending...
I20260924 05:04:12.193979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4809
I20260924 05:04:12.193992 4479 ts_manager.cc:194] Registered new tserver with Master: 0597242acde7490e86e300678938cfae (127.0.98.3:38269)
I20260924 05:04:12.194073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:12.194239 392 external_mini_cluster.cc:1428] Setting key 2f6b1367aa1a5cae69b9af549b52922c
I20260924 05:04:12.194393 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60589
I20260924 05:04:12.195029 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:04:12.201242 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.196217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.98.62:33849)
0924 05:04:12.196267 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.196269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.196348 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:04:12.196392 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.196393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.196400 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.196440 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.196916 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.197358 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.198073 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.198282 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.198326 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.199305 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.199306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.199306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:12.199312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.200465 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:12.200723 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:12.200724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:12.200724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:12.200772 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:12.200876 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:12.200876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:12.200876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:12.200986 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.201050 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.201118 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.201177 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:04:12.202471 4479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:12.203890 4479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:12.209613 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.205738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45105 (local address 127.0.0.1:52826)
0924 05:04:12.206173 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.206180 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.206218 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.206366 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.206366 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.206368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:12.206414 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.206415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.207372 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.207373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.208589 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.208592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.209419 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.209420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.209471 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.209474 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.209521 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.209552 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":74,"threads_started":1}
I20260924 05:04:12.209652 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.205951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.0.98.2:45105)
0924 05:04:12.206120 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.206125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.206249 (+ 124us) server_negotiation.cc:407] Connection header received
0924 05:04:12.206288 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.206290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.206304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.206363 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:12.206540 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.207285 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.208658 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.209347 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.209414 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.209468 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.209531 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.209570 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260924 05:04:12.209854 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.205737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34311 (local address 127.0.0.1:44054)
0924 05:04:12.206069 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.206078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.206113 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.206443 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.206444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.206446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:12.206479 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.206480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.207655 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.207656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.208513 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.208515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.209503 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.209504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.209574 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.209578 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.209625 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.209807 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":48,"threads_started":1}
I20260924 05:04:12.210062 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.206067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.98.1:34311)
0924 05:04:12.206236 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.206239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.206248 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:12.206280 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.206281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.206295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.206370 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:12.206547 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.207582 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.208606 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.209347 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.209414 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.209468 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.209950 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.209988 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260924 05:04:12.210350 4738 tablet_service.cc:1467] Processing CreateTablet for tablet d361294327b44d88be61d2e81a8aa3b6 (DEFAULT_TABLE table=test-table [id=7160e612c37d489ea45342d7203b83d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:12.210659 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d361294327b44d88be61d2e81a8aa3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.210632 4605 tablet_service.cc:1467] Processing CreateTablet for tablet d361294327b44d88be61d2e81a8aa3b6 (DEFAULT_TABLE table=test-table [id=7160e612c37d489ea45342d7203b83d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:12.210949 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d361294327b44d88be61d2e81a8aa3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.213019 4951 tablet_bootstrap.cc:492] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Bootstrap starting.
I20260924 05:04:12.209554 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.205730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38269 (local address 127.0.0.1:60394)
0924 05:04:12.205883 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.205921 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.205980 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.206252 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.206255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.206257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:12.206320 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.206321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.207564 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.207564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.208617 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.208618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.209305 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.209306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.209364 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.209372 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.209430 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.209480 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":49,"threads_started":1}
I20260924 05:04:12.213158 4952 tablet_bootstrap.cc:492] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3: Bootstrap starting.
I20260924 05:04:12.213354 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.205879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.3:38269)
0924 05:04:12.206020 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.206024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.206034 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:12.206070 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.206072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.206086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.206166 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:12.206393 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.207474 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.208702 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.209164 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.209220 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.209276 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.209590 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.209642 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 05:04:12.213784 4952 tablet_bootstrap.cc:654] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.213958 4951 tablet_bootstrap.cc:654] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.214001 4871 tablet_service.cc:1467] Processing CreateTablet for tablet d361294327b44d88be61d2e81a8aa3b6 (DEFAULT_TABLE table=test-table [id=7160e612c37d489ea45342d7203b83d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:12.214321 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d361294327b44d88be61d2e81a8aa3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.214334 4952 log.cc:824] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.214427 4951 log.cc:824] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.215045 4952 tablet_bootstrap.cc:492] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3: No bootstrap required, opened a new log
I20260924 05:04:12.215067 4951 tablet_bootstrap.cc:492] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: No bootstrap required, opened a new log
I20260924 05:04:12.215116 4951 ts_tablet_manager.cc:1397] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:12.215119 4952 ts_tablet_manager.cc:1397] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:04:12.216656 4955 tablet_bootstrap.cc:492] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Bootstrap starting.
I20260924 05:04:12.216981 4952 raft_consensus.cc:348] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.217154 4952 raft_consensus.cc:374] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.217200 4952 raft_consensus.cc:729] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbee93eb09c845f9b3444262364a19a3, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.217108 4951 raft_consensus.cc:348] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.217270 4951 raft_consensus.cc:374] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.217305 4951 raft_consensus.cc:729] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e38c37f2cdf46c6915f199f52e927fb, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.217311 4952 consensus_queue.cc:260] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.217347 4955 tablet_bootstrap.cc:654] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.217398 4951 consensus_queue.cc:260] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.217573 4952 ts_tablet_manager.cc:1428] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:04:12.217643 4951 ts_tablet_manager.cc:1428] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:04:12.217738 4671 heartbeater.cc:503] Master 127.0.98.62:33849 was elected leader, sending a full tablet report...
I20260924 05:04:12.217767 4804 heartbeater.cc:503] Master 127.0.98.62:33849 was elected leader, sending a full tablet report...
I20260924 05:04:12.217911 4955 log.cc:824] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.218823 4955 tablet_bootstrap.cc:492] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: No bootstrap required, opened a new log
I20260924 05:04:12.218894 4955 ts_tablet_manager.cc:1397] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:04:12.220148 4955 raft_consensus.cc:348] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.220273 4955 raft_consensus.cc:374] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.220293 4955 raft_consensus.cc:729] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0597242acde7490e86e300678938cfae, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.220379 4955 consensus_queue.cc:260] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.220645 4955 ts_tablet_manager.cc:1428] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:04:12.220649 4937 heartbeater.cc:503] Master 127.0.98.62:33849 was elected leader, sending a full tablet report...
W20260924 05:04:12.297314 4805 tablet.cc:2369] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:12.360163 4956 raft_consensus.cc:482] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:12.360284 4956 raft_consensus.cc:504] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.360637 4956 leader_election.cc:288] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0597242acde7490e86e300678938cfae (127.0.98.3:38269), 9e38c37f2cdf46c6915f199f52e927fb (127.0.98.2:45105)
I20260924 05:04:12.364288 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.360893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57685 (local address 127.0.98.2:45105)
0924 05:04:12.360951 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.360953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.361056 (+ 103us) server_negotiation.cc:407] Connection header received
0924 05:04:12.361115 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.361115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.361125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.361160 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:12.361267 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.361968 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.363386 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.363946 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.364019 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.364027 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.364182 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.364218 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:04:12.364288 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.360887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54099 (local address 127.0.98.3:38269)
0924 05:04:12.360929 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.360930 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.360962 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:04:12.360988 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.360989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.360999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.361043 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:12.361179 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.361966 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.363387 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.363946 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.364018 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.364027 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.364182 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.364218 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:12.364308 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.360801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38269 (local address 127.0.98.1:54099)
0924 05:04:12.360858 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.360869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.360912 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.361050 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.361051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.361053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:12.361123 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.361125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.362036 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.362037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.363306 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.363309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.364020 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.364021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.364117 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.364121 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.364173 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.364218 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":37}
I20260924 05:04:12.364312 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.360801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45105 (local address 127.0.98.1:57685)
0924 05:04:12.360996 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.361004 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.361061 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.361162 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.361162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.361163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:12.361212 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.361213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.362036 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.362036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.363306 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:12.363309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.364020 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.364021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:12.364117 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.364121 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.364173 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.364218 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":72,"thread_start_us":107,"threads_started":1}
I20260924 05:04:12.364641 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d361294327b44d88be61d2e81a8aa3b6" candidate_uuid: "fbee93eb09c845f9b3444262364a19a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0597242acde7490e86e300678938cfae" is_pre_election: true
I20260924 05:04:12.364797 4891 raft_consensus.cc:2454] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbee93eb09c845f9b3444262364a19a3 in term 0.
I20260924 05:04:12.364785 4758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d361294327b44d88be61d2e81a8aa3b6" candidate_uuid: "fbee93eb09c845f9b3444262364a19a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e38c37f2cdf46c6915f199f52e927fb" is_pre_election: true
I20260924 05:04:12.364900 4758 raft_consensus.cc:2454] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbee93eb09c845f9b3444262364a19a3 in term 0.
I20260924 05:04:12.365015 4560 leader_election.cc:302] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0597242acde7490e86e300678938cfae, fbee93eb09c845f9b3444262364a19a3; no voters:
I20260924 05:04:12.365149 4956 raft_consensus.cc:2781] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:12.365207 4956 raft_consensus.cc:482] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:12.365223 4956 raft_consensus.cc:3037] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:12.365927 4956 raft_consensus.cc:504] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.366044 4956 leader_election.cc:288] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [CANDIDATE]: Term 1 election: Requested vote from peers 0597242acde7490e86e300678938cfae (127.0.98.3:38269), 9e38c37f2cdf46c6915f199f52e927fb (127.0.98.2:45105)
I20260924 05:04:12.366209 4758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d361294327b44d88be61d2e81a8aa3b6" candidate_uuid: "fbee93eb09c845f9b3444262364a19a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e38c37f2cdf46c6915f199f52e927fb"
I20260924 05:04:12.366214 4891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d361294327b44d88be61d2e81a8aa3b6" candidate_uuid: "fbee93eb09c845f9b3444262364a19a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0597242acde7490e86e300678938cfae"
I20260924 05:04:12.366274 4758 raft_consensus.cc:3037] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:12.366279 4891 raft_consensus.cc:3037] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:12.366947 4758 raft_consensus.cc:2454] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbee93eb09c845f9b3444262364a19a3 in term 1.
I20260924 05:04:12.366969 4891 raft_consensus.cc:2454] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbee93eb09c845f9b3444262364a19a3 in term 1.
I20260924 05:04:12.367094 4559 leader_election.cc:302] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e38c37f2cdf46c6915f199f52e927fb, fbee93eb09c845f9b3444262364a19a3; no voters:
I20260924 05:04:12.367194 4956 raft_consensus.cc:2781] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:12.367312 4956 raft_consensus.cc:686] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [term 1 LEADER]: Becoming Leader. State: Replica: fbee93eb09c845f9b3444262364a19a3, State: Running, Role: LEADER
I20260924 05:04:12.367385 4956 consensus_queue.cc:237] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } }
I20260924 05:04:12.368100 4481 catalog_manager.cc:5495] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbee93eb09c845f9b3444262364a19a3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fbee93eb09c845f9b3444262364a19a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbee93eb09c845f9b3444262364a19a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:12.388707 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.386577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.0.98.1:34311)
0924 05:04:12.386621 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.386623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.386631 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:12.386684 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.386685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.386703 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.386759 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:12.386855 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.387363 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.388080 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:12.388271 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:12.388334 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:12.388448 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:12.388548 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:12.388605 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.388616 (+ 11us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.388633 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:12.390475 4891 raft_consensus.cc:1261] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 1 FOLLOWER]: Refusing update from remote peer fbee93eb09c845f9b3444262364a19a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:12.390475 4758 raft_consensus.cc:1261] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 1 FOLLOWER]: Refusing update from remote peer fbee93eb09c845f9b3444262364a19a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:12.390838 4961 consensus_queue.cc:1035] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e38c37f2cdf46c6915f199f52e927fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:12.390975 4956 consensus_queue.cc:1035] T d361294327b44d88be61d2e81a8aa3b6 P fbee93eb09c845f9b3444262364a19a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0597242acde7490e86e300678938cfae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:12.394210 4965 mvcc.cc:205] Tried to move back new op lower bound from 7332766729788772352 to 7332766729697247232. Current Snapshot: MvccSnapshot[applied={T|T < 7332766729788772352}]
I20260924 05:04:12.395506 4479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:12.395591 4479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:12.396234 4479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e432cc4e519d4d22b9a17948c0b80af5: Sending DeleteTablet for 3 replicas of tablet d361294327b44d88be61d2e81a8aa3b6
I20260924 05:04:12.396574 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet d361294327b44d88be61d2e81a8aa3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60394
I20260924 05:04:12.396641 4738 tablet_service.cc:1514] Processing DeleteTablet for tablet d361294327b44d88be61d2e81a8aa3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52826
I20260924 05:04:12.396770 4605 tablet_service.cc:1514] Processing DeleteTablet for tablet d361294327b44d88be61d2e81a8aa3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44054
I20260924 05:04:12.396868 4973 tablet_replica.cc:330] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: stopping tablet replica
I20260924 05:04:12.396953 4973 raft_consensus.cc:2229] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:12.397048 4973 raft_consensus.cc:2258] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:12.397060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4543
I20260924 05:04:12.399276 4973 ts_tablet_manager.cc:1905] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:12.400619 4973 ts_tablet_manager.cc:1918] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:12.400671 4973 log.cc:1196] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/d361294327b44d88be61d2e81a8aa3b6
I20260924 05:04:12.400887 4973 ts_tablet_manager.cc:1939] T d361294327b44d88be61d2e81a8aa3b6 P 0597242acde7490e86e300678938cfae: Deleting consensus metadata
W20260924 05:04:12.401043 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34311 (error 108)
I20260924 05:04:12.401237 4467 catalog_manager.cc:4841] TS 0597242acde7490e86e300678938cfae (127.0.98.3:38269): tablet d361294327b44d88be61d2e81a8aa3b6 (table test-table [id=7160e612c37d489ea45342d7203b83d2]) successfully deleted
I20260924 05:04:12.401275 4974 tablet_replica.cc:330] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: stopping tablet replica
I20260924 05:04:12.401353 4974 raft_consensus.cc:2229] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:12.401463 4974 raft_consensus.cc:2258] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 05:04:12.401703 4947 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:12.401431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34311 (local address 127.0.0.1:44084)
0924 05:04:12.401471 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.401482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.401525 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.401665 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 05:04:12.401822 4465 catalog_manager.cc:4584] TS fbee93eb09c845f9b3444262364a19a3 (127.0.98.1:34311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d361294327b44d88be61d2e81a8aa3b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:04:12.401876 4974 ts_tablet_manager.cc:1905] T d361294327b44d88be61d2e81a8aa3b6 P 9e38c37f2cdf46c6915f199f52e927fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:12.401894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4676
I20260924 05:04:12.405500 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.405411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45105 (local address 127.0.0.1:52836)
0924 05:04:12.405448 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.405479 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:04:12.406198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4809
I20260924 05:04:12.410706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4450
2026-09-24T05:04:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1859 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T05:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:12Z Disabled control of system clock
I20260924 05:04:12.436133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35089
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:04:12.513021 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:12.513219 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:12.513238 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:12.514371 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:12.514412 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:12.514423 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:12.514434 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:12.514446 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:12.515779 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:12.516000 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:12.516263 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:12.520596 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.520604 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.520624 4980 server_base.cc:1034] running on GCE node
W20260924 05:04:12.520606 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.521049 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:12.521323 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:12.522456 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790226252522438 us; error 33 us; skew 500 ppm
I20260924 05:04:12.523593 4980 webserver.cc:460] Webserver started at http://127.0.98.62:35441/ using document root <none> and password file <none>
I20260924 05:04:12.523808 4980 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:12.523854 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:12.523975 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:12.525035 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "8c84dc2c72a94656bd17f8df0b1017c4"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "1bdd596ff07069e03024e3589b18ee64"
server_key_iv: "7e48259025a311d8bee4e3915243e45b"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.525373 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c84dc2c72a94656bd17f8df0b1017c4"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "1bdd596ff07069e03024e3589b18ee64"
server_key_iv: "7e48259025a311d8bee4e3915243e45b"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.526640 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:12.527290 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.527422 4980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:12.527493 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "8c84dc2c72a94656bd17f8df0b1017c4"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "1bdd596ff07069e03024e3589b18ee64"
server_key_iv: "7e48259025a311d8bee4e3915243e45b"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.527568 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:12.530911 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:12.531155 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:12.531267 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:12.534888 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35089
I20260924 05:04:12.534929 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35089 every 8 connection(s)
I20260924 05:04:12.535215 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:12.535840 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.538581 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4: Bootstrap starting.
I20260924 05:04:12.539091 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.539317 5047 log.cc:824] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.539638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 4980
I20260924 05:04:12.539738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:12.539915 392 external_mini_cluster.cc:1428] Setting key 31f77345da5a43ca1a0ec972b132c44e
I20260924 05:04:12.539984 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4: No bootstrap required, opened a new log
I20260924 05:04:12.542033 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35089 } }
I20260924 05:04:12.542191 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.542227 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c84dc2c72a94656bd17f8df0b1017c4, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.542374 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35089 } }
I20260924 05:04:12.542439 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.540398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.0.98.62:35089)
0924 05:04:12.540596 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.540603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.540626 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:04:12.541180 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.541187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.541224 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.541333 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.542019 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.542022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.542023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.542026 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.542071 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.542082 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.542133 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.542218 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.542276 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260924 05:04:12.542459 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:12.542480 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:12.542518 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:12.543184 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35089 } }
I20260924 05:04:12.543286 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c84dc2c72a94656bd17f8df0b1017c4; no voters:
I20260924 05:04:12.543427 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:12.543452 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:12.543563 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [term 1 LEADER]: Becoming Leader. State: Replica: 8c84dc2c72a94656bd17f8df0b1017c4, State: Running, Role: LEADER
I20260924 05:04:12.543649 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35089 } }
I20260924 05:04:12.543673 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:12.543989 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c84dc2c72a94656bd17f8df0b1017c4. Latest consensus state: current_term: 1 leader_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35089 } } }
I20260924 05:04:12.543999 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c84dc2c72a94656bd17f8df0b1017c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35089 } } }
I20260924 05:04:12.544195 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:12.544339 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:12.544505 5061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:12.544940 5061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:12.546222 5061 catalog_manager.cc:1347] Generated new cluster ID: 3d0ad45836ee4de49e2c2f664c150d14
I20260924 05:04:12.546270 5061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:12.556769 5061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:12.556906 5061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:12.560810 5061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4: Generated new TSK 0
I20260924 05:04:12.560972 5061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:12.567939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35089
--builtin_ntp_servers=127.0.98.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:04:12.645747 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:12.645941 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:12.645961 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:12.647089 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:12.647126 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:12.647149 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:12.648432 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:12.648655 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:12.648918 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:12.653313 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.653425 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.653644 5071 server_base.cc:1034] running on GCE node
W20260924 05:04:12.653319 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.653891 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:12.654109 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:12.655246 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790226252655234 us; error 32 us; skew 500 ppm
I20260924 05:04:12.661441 5071 webserver.cc:460] Webserver started at http://127.0.98.1:44065/ using document root <none> and password file <none>
I20260924 05:04:12.661665 5071 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:12.661732 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:12.661860 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:12.662863 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a38b1989fc647ebab16c260ffce5614"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "fcf4fa7045e0997827036f26a49d1053"
server_key_iv: "7848537b3732cd3312a47652ddf23a3a"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.663177 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a38b1989fc647ebab16c260ffce5614"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "fcf4fa7045e0997827036f26a49d1053"
server_key_iv: "7848537b3732cd3312a47652ddf23a3a"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.664324 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:04:12.664912 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.665079 5071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:12.665159 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "5a38b1989fc647ebab16c260ffce5614"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "fcf4fa7045e0997827036f26a49d1053"
server_key_iv: "7848537b3732cd3312a47652ddf23a3a"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.665217 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:12.668316 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:12.668536 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:12.668671 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:12.668841 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:12.669113 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:12.669144 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.669168 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:12.669193 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.674343 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46151
I20260924 05:04:12.674392 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46151 every 8 connection(s)
I20260924 05:04:12.674645 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:12.677182 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.675208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53555 (local address 127.0.98.62:35089)
0924 05:04:12.675253 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.675254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.675441 (+ 187us) server_negotiation.cc:407] Connection header received
0924 05:04:12.675873 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.675874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.675881 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.675931 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.676954 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.676955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.676956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.676961 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.676976 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.676980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.677028 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.677105 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.677118 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:04:12.677343 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.675191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35089 (local address 127.0.98.1:53555)
0924 05:04:12.675339 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.675369 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.675787 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.675994 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.675998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.676008 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.676817 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:12.676818 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.676844 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:12.676870 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.677032 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.677036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.677039 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.677110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.677167 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260924 05:04:12.677747 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35089
I20260924 05:04:12.677836 5199 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:12.677964 5199 heartbeater.cc:511] Master 127.0.98.62:35089 requested a full tablet report, sending...
I20260924 05:04:12.678375 5011 ts_manager.cc:194] Registered new tserver with Master: 5a38b1989fc647ebab16c260ffce5614 (127.0.98.1:46151)
I20260924 05:04:12.682397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5071
I20260924 05:04:12.682494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:12.682662 392 external_mini_cluster.cc:1428] Setting key d6ded05a6fcab3520d29450c8eb73a79
I20260924 05:04:12.683357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35089
--builtin_ntp_servers=127.0.98.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:04:12.760743 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:12.760934 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:12.760963 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:12.762084 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:12.762120 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:12.762152 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:12.763662 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:12.763880 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:12.764123 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:12.768616 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.768595 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.768594 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.768886 5202 server_base.cc:1034] running on GCE node
I20260924 05:04:12.769068 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:12.769251 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:12.770375 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790226252770363 us; error 28 us; skew 500 ppm
I20260924 05:04:12.771492 5202 webserver.cc:460] Webserver started at http://127.0.98.2:46741/ using document root <none> and password file <none>
I20260924 05:04:12.771672 5202 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:12.771711 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:12.771801 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:12.772751 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "390ab2c51bc4443d90ae5ef63e625851"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "474e4341967920f69676c24b7b997559"
server_key_iv: "514cad3e748844606adfa96a93cc9ddd"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.773023 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "390ab2c51bc4443d90ae5ef63e625851"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "474e4341967920f69676c24b7b997559"
server_key_iv: "514cad3e748844606adfa96a93cc9ddd"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.774158 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:04:12.774731 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.774861 5202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:12.774933 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "390ab2c51bc4443d90ae5ef63e625851"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "474e4341967920f69676c24b7b997559"
server_key_iv: "514cad3e748844606adfa96a93cc9ddd"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.774983 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:12.778015 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:12.778206 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:12.778290 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:12.778422 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:12.778650 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:12.778678 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.778695 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:12.778712 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.784454 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33543
I20260924 05:04:12.784765 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33543 every 8 connection(s)
I20260924 05:04:12.784822 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:12.787207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5202
I20260924 05:04:12.787307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:12.787492 392 external_mini_cluster.cc:1428] Setting key 6d64696bbc530adcbc5ce86151b35f73
I20260924 05:04:12.788079 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.785825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36083 (local address 127.0.98.62:35089)
0924 05:04:12.785874 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.785876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.785887 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:12.786447 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.786448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.786454 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.786517 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.787772 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.787773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.787773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.787779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.787814 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.787819 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.787876 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.787986 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.788003 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:04:12.788439 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.785511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35089 (local address 127.0.98.2:36083)
0924 05:04:12.785757 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.785801 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.786337 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.786589 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.786593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.786603 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.787627 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:12.787628 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.787664 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:12.787697 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.787887 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.787893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.787896 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.788160 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.788226 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260924 05:04:12.790202 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35089
I20260924 05:04:12.790292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35089
--builtin_ntp_servers=127.0.98.20:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 05:04:12.790321 5330 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:12.790477 5330 heartbeater.cc:511] Master 127.0.98.62:35089 requested a full tablet report, sending...
I20260924 05:04:12.790897 5011 ts_manager.cc:194] Registered new tserver with Master: 390ab2c51bc4443d90ae5ef63e625851 (127.0.98.2:33543)
W20260924 05:04:12.879560 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:12.879747 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:12.879773 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:12.880882 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:12.880919 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:12.880952 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:12.882257 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:12.882471 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:12.882711 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:12.887096 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.887121 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:12.887097 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:12.887416 5333 server_base.cc:1034] running on GCE node
I20260924 05:04:12.887598 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:12.887779 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:12.888916 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790226252888896 us; error 30 us; skew 500 ppm
I20260924 05:04:12.890086 5333 webserver.cc:460] Webserver started at http://127.0.98.3:40135/ using document root <none> and password file <none>
I20260924 05:04:12.890288 5333 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:12.890336 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:12.890456 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:12.891456 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "dd24183010684a859b61ab1d48f7067c"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "f8bd7deac53700f2ed03f794851eb37c"
server_key_iv: "024284aa3a29578182b64d6f55d0a43d"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.891788 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dd24183010684a859b61ab1d48f7067c"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "f8bd7deac53700f2ed03f794851eb37c"
server_key_iv: "024284aa3a29578182b64d6f55d0a43d"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.892910 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:12.893479 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.893625 5333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.893704 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "dd24183010684a859b61ab1d48f7067c"
format_stamp: "Formatted at 2026-09-24 05:04:12 on dist-test-slave-m01t"
server_key: "f8bd7deac53700f2ed03f794851eb37c"
server_key_iv: "024284aa3a29578182b64d6f55d0a43d"
server_key_version: "encryptionkey@0"
I20260924 05:04:12.893791 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:12.896864 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:12.897084 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:12.897194 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:12.897373 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:12.897627 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:12.897665 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.897689 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:12.897727 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:12.902740 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38901
I20260924 05:04:12.902791 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38901 every 8 connection(s)
I20260924 05:04:12.903043 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:12.905548 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.903616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35603 (local address 127.0.98.62:35089)
0924 05:04:12.903666 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.903668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.903846 (+ 178us) server_negotiation.cc:407] Connection header received
0924 05:04:12.904287 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.904288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.904295 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.904345 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.905324 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.905325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.905325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.905329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.905345 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.905349 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.905398 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.905469 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.905483 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:04:12.905753 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.903590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35089 (local address 127.0.98.3:35603)
0924 05:04:12.903745 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.903775 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.904195 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.904412 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.904416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.904425 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.905194 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:12.905194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.905220 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:12.905244 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.905402 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.905405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.905408 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.905469 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.905525 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260924 05:04:12.906136 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35089
I20260924 05:04:12.906210 5461 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:12.906334 5461 heartbeater.cc:511] Master 127.0.98.62:35089 requested a full tablet report, sending...
I20260924 05:04:12.906687 5011 ts_manager.cc:194] Registered new tserver with Master: dd24183010684a859b61ab1d48f7067c (127.0.98.3:38901)
I20260924 05:04:12.907160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5333
I20260924 05:04:12.907248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:12.907400 392 external_mini_cluster.cc:1428] Setting key d29757c0ef1d2ad8c729ddbeaf349956
I20260924 05:04:12.907994 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:04:12.910027 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.909116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.98.62:35089)
0924 05:04:12.909161 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.909162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.909287 (+ 125us) server_negotiation.cc:407] Connection header received
0924 05:04:12.909336 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.909337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.909341 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.909375 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.909875 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.909876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.909876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.909878 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.909888 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.909891 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.909925 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.909958 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.909967 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:04:12.911389 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:12.912781 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:12.915992 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.914598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33543 (local address 127.0.0.1:38748)
0924 05:04:12.914835 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.914872 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.915053 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.915179 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.915183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.915188 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.915618 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:12.915618 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.915633 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:12.915653 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.915844 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.915849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.915852 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.915912 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.915925 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260924 05:04:12.916018 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.914679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.0.98.2:33543)
0924 05:04:12.914830 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.914834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.914935 (+ 101us) server_negotiation.cc:407] Connection header received
0924 05:04:12.915100 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.915104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.915113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.915176 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.915716 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.915722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.915722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.915726 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.915760 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.915784 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.915841 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.915929 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.915945 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260924 05:04:12.916625 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.914776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.0.1:46674)
0924 05:04:12.914979 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.914987 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.915015 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.915594 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.915596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.915598 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.916264 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:12.916265 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.916270 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:12.916281 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.916492 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.916493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.916493 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.916536 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.916549 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":64,"threads_started":1}
I20260924 05:04:12.916656 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.914965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.1:46151)
0924 05:04:12.915131 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.915135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.915143 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:12.915166 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.915167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.915175 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.915238 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.916354 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.916362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.916362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.916367 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.916399 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.916422 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.916488 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.916552 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.916566 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260924 05:04:12.916846 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.914671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.98.3:38901)
0924 05:04:12.914901 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:12.914905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:12.914916 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:12.914980 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:12.914982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:12.914990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:12.915086 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:12.916401 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:12.916406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:12.916406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:12.916410 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:12.916436 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:12.916453 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:12.916501 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:12.916652 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 05:04:12.916666 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260924 05:04:12.916756 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 711963b3ba6446b1a68bc8741d6484cb (DEFAULT_TABLE table=test-table [id=5381b350ee284bd0901979895cca9a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:12.916950 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:12.914598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38901 (local address 127.0.0.1:41914)
0924 05:04:12.914877 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:12.914886 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:12.914929 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:12.915506 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:12.915507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:12.915509 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:12.915849 (+ 340us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:12.915849 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:12.915856 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:12.915866 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:12.916525 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:12.916527 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:12.916527 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 05:04:12.916569 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:12.916581 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260924 05:04:12.917085 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711963b3ba6446b1a68bc8741d6484cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.917569 5395 tablet_service.cc:1467] Processing CreateTablet for tablet 711963b3ba6446b1a68bc8741d6484cb (DEFAULT_TABLE table=test-table [id=5381b350ee284bd0901979895cca9a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:12.917892 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711963b3ba6446b1a68bc8741d6484cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.918215 5133 tablet_service.cc:1467] Processing CreateTablet for tablet 711963b3ba6446b1a68bc8741d6484cb (DEFAULT_TABLE table=test-table [id=5381b350ee284bd0901979895cca9a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:12.918430 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711963b3ba6446b1a68bc8741d6484cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:12.920190 5475 tablet_bootstrap.cc:492] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Bootstrap starting.
I20260924 05:04:12.920804 5476 tablet_bootstrap.cc:492] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Bootstrap starting.
I20260924 05:04:12.920954 5475 tablet_bootstrap.cc:654] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.921237 5477 tablet_bootstrap.cc:492] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614: Bootstrap starting.
I20260924 05:04:12.921288 5475 log.cc:824] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.921357 5476 tablet_bootstrap.cc:654] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.921662 5476 log.cc:824] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.921852 5477 tablet_bootstrap.cc:654] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:12.922035 5475 tablet_bootstrap.cc:492] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: No bootstrap required, opened a new log
I20260924 05:04:12.922104 5475 ts_tablet_manager.cc:1397] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:12.922153 5477 log.cc:824] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:12.922402 5476 tablet_bootstrap.cc:492] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: No bootstrap required, opened a new log
I20260924 05:04:12.922470 5476 ts_tablet_manager.cc:1397] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:04:12.922827 5477 tablet_bootstrap.cc:492] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614: No bootstrap required, opened a new log
I20260924 05:04:12.922928 5477 ts_tablet_manager.cc:1397] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:12.924320 5476 raft_consensus.cc:348] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:12.924449 5476 raft_consensus.cc:374] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.924479 5476 raft_consensus.cc:729] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd24183010684a859b61ab1d48f7067c, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.924548 5475 raft_consensus.cc:348] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:12.924641 5476 consensus_queue.cc:260] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:12.924717 5475 raft_consensus.cc:374] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.924757 5475 raft_consensus.cc:729] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390ab2c51bc4443d90ae5ef63e625851, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.924887 5476 ts_tablet_manager.cc:1428] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:04:12.924891 5475 consensus_queue.cc:260] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:12.924997 5461 heartbeater.cc:503] Master 127.0.98.62:35089 was elected leader, sending a full tablet report...
I20260924 05:04:12.925143 5475 ts_tablet_manager.cc:1428] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:04:12.925334 5330 heartbeater.cc:503] Master 127.0.98.62:35089 was elected leader, sending a full tablet report...
I20260924 05:04:12.925443 5477 raft_consensus.cc:348] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:12.925588 5477 raft_consensus.cc:374] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:12.925628 5477 raft_consensus.cc:729] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a38b1989fc647ebab16c260ffce5614, State: Initialized, Role: FOLLOWER
I20260924 05:04:12.925758 5477 consensus_queue.cc:260] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:12.925994 5477 ts_tablet_manager.cc:1428] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 05:04:12.926111 5199 heartbeater.cc:503] Master 127.0.98.62:35089 was elected leader, sending a full tablet report...
I20260924 05:04:13.015106 5481 raft_consensus.cc:482] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:13.015236 5481 raft_consensus.cc:504] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:13.015609 5481 leader_election.cc:288] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 390ab2c51bc4443d90ae5ef63e625851 (127.0.98.2:33543), 5a38b1989fc647ebab16c260ffce5614 (127.0.98.1:46151)
I20260924 05:04:13.016685 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.015834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37857 (local address 127.0.98.2:33543)
0924 05:04:13.015879 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.015881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.015892 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:13.015941 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.015942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.015948 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.016007 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.016488 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.016490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.016490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.016494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.016513 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.016517 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.016570 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.016617 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.016628 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:04:13.016693 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.015766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33543 (local address 127.0.98.3:37857)
0924 05:04:13.015832 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.015844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:13.015885 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:13.016017 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:13.016018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:13.016020 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:13.016393 (+ 373us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:13.016393 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:13.016400 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:13.016411 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:13.016574 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:13.016575 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:13.016575 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 05:04:13.016618 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:04:13.016630 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 05:04:13.017062 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711963b3ba6446b1a68bc8741d6484cb" candidate_uuid: "dd24183010684a859b61ab1d48f7067c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390ab2c51bc4443d90ae5ef63e625851" is_pre_election: true
I20260924 05:04:13.017141 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.015775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.98.3:50297)
0924 05:04:13.015999 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.016008 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:13.016059 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:13.016392 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:13.016393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:13.016395 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:13.016873 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:13.016874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:13.016880 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:13.016892 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:13.017037 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:13.017038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:13.017038 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 05:04:13.017068 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 05:04:13.017078 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":57,"thread_start_us":99,"threads_started":1}
I20260924 05:04:13.017150 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.015849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50297 (local address 127.0.98.1:46151)
0924 05:04:13.015885 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.015886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.016074 (+ 188us) server_negotiation.cc:407] Connection header received
0924 05:04:13.016104 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.016105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.016112 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.016166 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.016957 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.016958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.016958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.016963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.016979 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.016984 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.017039 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.017079 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.017091 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:13.017221 5284 raft_consensus.cc:2454] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd24183010684a859b61ab1d48f7067c in term 0.
I20260924 05:04:13.017446 5349 leader_election.cc:302] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390ab2c51bc4443d90ae5ef63e625851, dd24183010684a859b61ab1d48f7067c; no voters:
I20260924 05:04:13.017576 5481 raft_consensus.cc:2781] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:13.017623 5481 raft_consensus.cc:482] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:13.017570 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711963b3ba6446b1a68bc8741d6484cb" candidate_uuid: "dd24183010684a859b61ab1d48f7067c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a38b1989fc647ebab16c260ffce5614" is_pre_election: true
I20260924 05:04:13.017650 5481 raft_consensus.cc:3037] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.017732 5153 raft_consensus.cc:2454] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd24183010684a859b61ab1d48f7067c in term 0.
I20260924 05:04:13.018400 5481 raft_consensus.cc:504] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:13.018504 5481 leader_election.cc:288] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [CANDIDATE]: Term 1 election: Requested vote from peers 390ab2c51bc4443d90ae5ef63e625851 (127.0.98.2:33543), 5a38b1989fc647ebab16c260ffce5614 (127.0.98.1:46151)
I20260924 05:04:13.018637 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711963b3ba6446b1a68bc8741d6484cb" candidate_uuid: "dd24183010684a859b61ab1d48f7067c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390ab2c51bc4443d90ae5ef63e625851"
I20260924 05:04:13.018656 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711963b3ba6446b1a68bc8741d6484cb" candidate_uuid: "dd24183010684a859b61ab1d48f7067c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a38b1989fc647ebab16c260ffce5614"
I20260924 05:04:13.018713 5153 raft_consensus.cc:3037] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.018715 5284 raft_consensus.cc:3037] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.019373 5153 raft_consensus.cc:2454] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd24183010684a859b61ab1d48f7067c in term 1.
I20260924 05:04:13.019379 5284 raft_consensus.cc:2454] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd24183010684a859b61ab1d48f7067c in term 1.
I20260924 05:04:13.019506 5349 leader_election.cc:302] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390ab2c51bc4443d90ae5ef63e625851, dd24183010684a859b61ab1d48f7067c; no voters:
I20260924 05:04:13.019604 5481 raft_consensus.cc:2781] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:13.019739 5481 raft_consensus.cc:686] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 1 LEADER]: Becoming Leader. State: Replica: dd24183010684a859b61ab1d48f7067c, State: Running, Role: LEADER
I20260924 05:04:13.019820 5481 consensus_queue.cc:237] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260924 05:04:13.020601 5010 catalog_manager.cc:5495] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c reported cstate change: term changed from 0 to 1, leader changed from <none> to dd24183010684a859b61ab1d48f7067c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dd24183010684a859b61ab1d48f7067c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24183010684a859b61ab1d48f7067c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:13.031803 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.031003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.0.98.3:38901)
0924 05:04:13.031049 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.031051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.031082 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:04:13.031114 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.031115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.031128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.031173 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.031662 (+ 489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.031663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.031663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.031665 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.031675 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.031678 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.031711 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.031741 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.031750 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:04:13.033455 5284 raft_consensus.cc:1261] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 1 FOLLOWER]: Refusing update from remote peer dd24183010684a859b61ab1d48f7067c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:13.033486 5153 raft_consensus.cc:1261] T 711963b3ba6446b1a68bc8741d6484cb P 5a38b1989fc647ebab16c260ffce5614 [term 1 FOLLOWER]: Refusing update from remote peer dd24183010684a859b61ab1d48f7067c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:13.033818 5481 consensus_queue.cc:1035] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a38b1989fc647ebab16c260ffce5614" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:13.033947 5485 consensus_queue.cc:1035] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [LEADER]: Connected to new peer: Peer: permanent_uuid: "390ab2c51bc4443d90ae5ef63e625851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:04:13.035374 5331 tablet.cc:2369] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:13.035953 5488 mvcc.cc:205] Tried to move back new op lower bound from 7332766732422615040 to 7332766732369793024. Current Snapshot: MvccSnapshot[applied={T|T < 7332766732422615040}]
I20260924 05:04:13.038233 5011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:13.038311 5011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:13.038973 5011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c84dc2c72a94656bd17f8df0b1017c4: Sending DeleteTablet for 3 replicas of tablet 711963b3ba6446b1a68bc8741d6484cb
I20260924 05:04:13.039265 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 711963b3ba6446b1a68bc8741d6484cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:13 UTC) from {username='slave'} at 127.0.0.1:38748
I20260924 05:04:13.039356 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet 711963b3ba6446b1a68bc8741d6484cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:13 UTC) from {username='slave'} at 127.0.0.1:41914
I20260924 05:04:13.039340 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet 711963b3ba6446b1a68bc8741d6484cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:13 UTC) from {username='slave'} at 127.0.0.1:46674
I20260924 05:04:13.039541 5498 tablet_replica.cc:330] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: stopping tablet replica
I20260924 05:04:13.039624 5498 raft_consensus.cc:2229] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:13.039716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5071
I20260924 05:04:13.039733 5498 raft_consensus.cc:2258] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:13.041152 5500 tablet_replica.cc:330] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: stopping tablet replica
I20260924 05:04:13.041244 5500 raft_consensus.cc:2229] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:13.041327 5500 raft_consensus.cc:2258] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:13.041558 5498 ts_tablet_manager.cc:1905] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:13.043226 5498 ts_tablet_manager.cc:1918] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:13.043340 5498 log.cc:1196] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/711963b3ba6446b1a68bc8741d6484cb
I20260924 05:04:13.043670 5498 ts_tablet_manager.cc:1939] T 711963b3ba6446b1a68bc8741d6484cb P 390ab2c51bc4443d90ae5ef63e625851: Deleting consensus metadata
I20260924 05:04:13.044106 4996 catalog_manager.cc:4841] TS 390ab2c51bc4443d90ae5ef63e625851 (127.0.98.2:33543): tablet 711963b3ba6446b1a68bc8741d6484cb (table test-table [id=5381b350ee284bd0901979895cca9a98]) successfully deleted
W20260924 05:04:13.044776 4997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46151 (error 108)
I20260924 05:04:13.045131 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.045022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.0.1:46682)
0924 05:04:13.045066 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.045101 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:04:13.045145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5202
I20260924 05:04:13.045817 5500 ts_tablet_manager.cc:1905] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:04:13.046455 4997 catalog_manager.cc:4584] TS 5a38b1989fc647ebab16c260ffce5614 (127.0.98.1:46151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 711963b3ba6446b1a68bc8741d6484cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46151: connect: Connection refused (error 111)
I20260924 05:04:13.047200 5500 ts_tablet_manager.cc:1918] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:13.047271 5500 log.cc:1196] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/711963b3ba6446b1a68bc8741d6484cb
I20260924 05:04:13.047508 5500 ts_tablet_manager.cc:1939] T 711963b3ba6446b1a68bc8741d6484cb P dd24183010684a859b61ab1d48f7067c: Deleting consensus metadata
I20260924 05:04:13.047832 4997 catalog_manager.cc:4841] TS dd24183010684a859b61ab1d48f7067c (127.0.98.3:38901): tablet 711963b3ba6446b1a68bc8741d6484cb (table test-table [id=5381b350ee284bd0901979895cca9a98]) successfully deleted
I20260924 05:04:13.050339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5333
I20260924 05:04:13.054793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 4980
2026-09-24T05:04:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (642 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 05:04:13.072289 392 mini_oidc.cc:152] Starting JWKS server
I20260924 05:04:13.073283 392 webserver.cc:460] Webserver started at https://127.0.0.1:39155/ using document root <none> and password file <none>
I20260924 05:04:13.073372 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 05:04:13.073529 392 webserver.cc:460] Webserver started at http://127.0.0.1:41619/ using document root <none> and password file <none>
2026-09-24T05:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:13Z Disabled control of system clock
I20260924 05:04:13.079902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44347
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39155/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/testchainca.cert with env {}
W20260924 05:04:13.157224 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:13.157424 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:13.157452 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:13.158632 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:13.158674 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:13.158689 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:13.158704 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:13.158720 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:04:13.158744 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:04:13.158761 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39155/jwks.json
I20260924 05:04:13.160252 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39155/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:13.160499 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:13.160686 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:13.164971 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:13.164996 5507 server_base.cc:1034] running on GCE node
W20260924 05:04:13.164971 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:13.165000 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:13.165427 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:13.165668 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:13.166819 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790226253166806 us; error 31 us; skew 500 ppm
I20260924 05:04:13.167939 5507 webserver.cc:460] Webserver started at http://127.0.98.62:42783/ using document root <none> and password file <none>
I20260924 05:04:13.168134 5507 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:13.168174 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:13.168259 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:13.169196 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "c890cd1d83c245d78926727700ed6fa4"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "398e957b44ddc8f8b6d7c1433728c289"
server_key_iv: "2b9dbaff0254d675e41716edf5720f83"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.169466 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "c890cd1d83c245d78926727700ed6fa4"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "398e957b44ddc8f8b6d7c1433728c289"
server_key_iv: "2b9dbaff0254d675e41716edf5720f83"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.170717 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:04:13.171350 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:13.171487 5507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:13.171557 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "c890cd1d83c245d78926727700ed6fa4"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "398e957b44ddc8f8b6d7c1433728c289"
server_key_iv: "2b9dbaff0254d675e41716edf5720f83"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.171625 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:13.183823 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:13.184070 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:13.184194 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:13.187573 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44347
I20260924 05:04:13.187608 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44347 every 8 connection(s)
I20260924 05:04:13.187883 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:13.188791 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:13.191075 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4: Bootstrap starting.
I20260924 05:04:13.191601 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:13.191823 5574 log.cc:824] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:13.192394 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4: No bootstrap required, opened a new log
I20260924 05:04:13.193234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5507
I20260924 05:04:13.193327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:13.193527 392 external_mini_cluster.cc:1428] Setting key 13a4bf516ef7e2d29cfdeb691d02e8a3
I20260924 05:04:13.194339 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c890cd1d83c245d78926727700ed6fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260924 05:04:13.194504 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:13.194554 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c890cd1d83c245d78926727700ed6fa4, State: Initialized, Role: FOLLOWER
I20260924 05:04:13.194681 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c890cd1d83c245d78926727700ed6fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260924 05:04:13.194762 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:13.194788 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:13.194832 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.195456 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c890cd1d83c245d78926727700ed6fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260924 05:04:13.195566 5574 leader_election.cc:302] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c890cd1d83c245d78926727700ed6fa4; no voters:
I20260924 05:04:13.195717 5574 leader_election.cc:288] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:13.195748 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:13.195884 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [term 1 LEADER]: Becoming Leader. State: Replica: c890cd1d83c245d78926727700ed6fa4, State: Running, Role: LEADER
I20260924 05:04:13.195940 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:13.195971 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c890cd1d83c245d78926727700ed6fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260924 05:04:13.196260 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c890cd1d83c245d78926727700ed6fa4. Latest consensus state: current_term: 1 leader_uuid: "c890cd1d83c245d78926727700ed6fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c890cd1d83c245d78926727700ed6fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260924 05:04:13.196327 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.196465 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c890cd1d83c245d78926727700ed6fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c890cd1d83c245d78926727700ed6fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260924 05:04:13.196520 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.196767 5585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:13.197259 5585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:13.199014 5585 catalog_manager.cc:1347] Generated new cluster ID: 2f64f4b1c3d8419eb83e73788e2b516a
I20260924 05:04:13.199066 5585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:13.199723 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.194027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.0.98.62:44347)
0924 05:04:13.194248 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.194257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.194287 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:04:13.194395 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.194402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.194467 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.194632 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.194902 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.196610 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.197620 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.198177 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.198291 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.199054 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.199058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.199058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.199254 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.199293 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.199306 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.199362 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.199464 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.199532 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260924 05:04:13.200522 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:04:13.206061 5585 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:13.206516 5585 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:13.212877 5585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c890cd1d83c245d78926727700ed6fa4: Generated new TSK 0
I20260924 05:04:13.213004 5585 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 05:04:13.236989 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 05:04:13.237098 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 05:04:13.237331 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.222590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.0.98.62:44347)
0924 05:04:13.222630 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.222632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.222721 (+ 89us) server_negotiation.cc:407] Connection header received
0924 05:04:13.222822 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.222823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.222832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.222884 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:04:13.223026 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.223498 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.224578 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.224748 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.224788 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.224990 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:04:13.237093 (+ 12103us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39155/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 05:04:13.237266 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39155/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:04:13.237388 5578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39155/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 05:04:13.237761 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:13.222485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.0.1:53552)
0924 05:04:13.222653 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.222665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:13.222735 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:13.222907 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:13.222907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:13.222910 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:04:13.222973 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.222975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.223567 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.223568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.224506 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.224510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.224808 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.224811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.224877 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.224899 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:04:13.237283 (+ 12384us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39155/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 05:04:13.237392 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39155/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44347: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39155/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 05:04:13.238106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5507
2026-09-24T05:04:13Z chronyd exiting
I20260924 05:04:13.272966 392 test_util.cc:175] -----------------------------------------------
I20260924 05:04:13.273036 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226228698807-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (201 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 05:04:13.273747 392 mini_oidc.cc:152] Starting JWKS server
I20260924 05:04:13.274385 392 webserver.cc:460] Webserver started at https://127.0.0.1:33581/ using document root <none> and password file <none>
I20260924 05:04:13.274423 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 05:04:13.274549 392 webserver.cc:460] Webserver started at http://127.0.0.1:36955/ using document root <none> and password file <none>
2026-09-24T05:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:13Z Disabled control of system clock
I20260924 05:04:13.281102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46019
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/testchainca.cert with env {}
W20260924 05:04:13.358549 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:13.358737 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:13.358757 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:13.359833 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:13.359864 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:13.359876 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:13.359887 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:13.359903 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:04:13.359915 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:04:13.359925 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33581/jwks.json
I20260924 05:04:13.361330 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:13.361508 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:13.361657 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:13.365967 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:13.365974 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:13.365994 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:13.366132 5610 server_base.cc:1034] running on GCE node
I20260924 05:04:13.366396 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:13.366648 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:13.367782 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790226253367768 us; error 26 us; skew 500 ppm
I20260924 05:04:13.368834 5610 webserver.cc:460] Webserver started at http://127.0.98.62:40997/ using document root <none> and password file <none>
I20260924 05:04:13.369027 5610 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:13.369072 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:13.369184 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:13.370205 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "34070feec02e4d268f5eb54b6362dbfe"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "d18f214e062d5a21ceff18541b698c4f"
server_key_iv: "53507eb51a47cd53485d674c4204798d"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.370515 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "34070feec02e4d268f5eb54b6362dbfe"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "d18f214e062d5a21ceff18541b698c4f"
server_key_iv: "53507eb51a47cd53485d674c4204798d"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.371613 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:13.372201 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:13.372344 5610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:13.372418 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "34070feec02e4d268f5eb54b6362dbfe"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "d18f214e062d5a21ceff18541b698c4f"
server_key_iv: "53507eb51a47cd53485d674c4204798d"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.372473 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:13.391019 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:13.391316 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:13.391443 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:13.395027 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46019
I20260924 05:04:13.395068 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46019 every 8 connection(s)
I20260924 05:04:13.395367 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:13.396238 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:13.398453 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe: Bootstrap starting.
I20260924 05:04:13.398964 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:13.399214 5677 log.cc:824] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:13.399791 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe: No bootstrap required, opened a new log
I20260924 05:04:13.401278 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070feec02e4d268f5eb54b6362dbfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260924 05:04:13.401404 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:13.401434 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34070feec02e4d268f5eb54b6362dbfe, State: Initialized, Role: FOLLOWER
I20260924 05:04:13.401561 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070feec02e4d268f5eb54b6362dbfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260924 05:04:13.401619 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:13.401648 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:13.401692 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.402179 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070feec02e4d268f5eb54b6362dbfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260924 05:04:13.402266 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34070feec02e4d268f5eb54b6362dbfe; no voters:
I20260924 05:04:13.402410 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:13.402444 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:13.402590 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [term 1 LEADER]: Becoming Leader. State: Replica: 34070feec02e4d268f5eb54b6362dbfe, State: Running, Role: LEADER
I20260924 05:04:13.402627 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:13.402681 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070feec02e4d268f5eb54b6362dbfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } }
I20260924 05:04:13.403005 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34070feec02e4d268f5eb54b6362dbfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070feec02e4d268f5eb54b6362dbfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } } }
I20260924 05:04:13.403091 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.403245 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34070feec02e4d268f5eb54b6362dbfe. Latest consensus state: current_term: 1 leader_uuid: "34070feec02e4d268f5eb54b6362dbfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070feec02e4d268f5eb54b6362dbfe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46019 } } }
I20260924 05:04:13.403291 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:13.403306 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.403867 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:13.404675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5610
I20260924 05:04:13.404774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:13.404963 392 external_mini_cluster.cc:1428] Setting key fba50b642c07700be4d5327e3143a665
I20260924 05:04:13.405345 5688 catalog_manager.cc:1347] Generated new cluster ID: 5935ba5bcde94cc6a3e34f330529d35a
I20260924 05:04:13.405392 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:13.409739 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.405340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.98.62:46019)
0924 05:04:13.405986 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.405993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.406014 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:04:13.406102 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.406108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.406167 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.406273 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.406457 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.407313 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.408047 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.408456 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.408557 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.409140 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.409144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.409144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.409300 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.409331 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.409341 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.409393 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.409467 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.409539 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":62,"threads_started":1}
I20260924 05:04:13.410496 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:04:13.455506 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:13.456432 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:13.459160 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34070feec02e4d268f5eb54b6362dbfe: Generated new TSK 0
I20260924 05:04:13.459362 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 05:04:13.477859 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 05:04:13.477937 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 05:04:13.478241 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.464557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.0.98.62:46019)
0924 05:04:13.464609 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.464611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.464724 (+ 113us) server_negotiation.cc:407] Connection header received
0924 05:04:13.464780 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.464781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.464794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.464854 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:04:13.464974 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.465447 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.466232 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.466419 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.466471 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.466622 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:04:13.478032 (+ 11410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 05:04:13.478174 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 05:04:13.478296 5700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 05:04:13.478338 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:13.464490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46019 (local address 127.0.0.1:42708)
0924 05:04:13.464661 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.464670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:13.464724 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:13.464859 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:13.464860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:13.464861 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:04:13.464917 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.464920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.465518 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.465519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.466160 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.466163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.466477 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.466481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.466537 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.466542 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:04:13.478145 (+ 11603us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 05:04:13.478251 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260924 05:04:13.478653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5610
2026-09-24T05:04:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 05:04:13.515219 392 mini_oidc.cc:152] Starting JWKS server
I20260924 05:04:13.515866 392 webserver.cc:460] Webserver started at https://127.0.0.1:40107/ using document root <none> and password file <none>
I20260924 05:04:13.515908 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 05:04:13.516057 392 webserver.cc:460] Webserver started at http://127.0.0.1:39161/ using document root <none> and password file <none>
2026-09-24T05:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:13Z Disabled control of system clock
I20260924 05:04:13.522943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44427
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44427
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40107/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 05:04:13.600967 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:13.601154 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:13.601173 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:13.602267 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:13.602299 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:13.602310 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:13.602320 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:13.602331 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:04:13.602351 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:04:13.602361 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40107/jwks.json
I20260924 05:04:13.603596 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40107/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:13.603770 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:13.603991 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:13.608202 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:13.608227 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:13.608232 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:13.608249 5713 server_base.cc:1034] running on GCE node
I20260924 05:04:13.608567 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:13.608831 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:13.609965 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790226253609953 us; error 23 us; skew 500 ppm
I20260924 05:04:13.610991 5713 webserver.cc:460] Webserver started at http://127.0.98.62:38589/ using document root <none> and password file <none>
I20260924 05:04:13.611191 5713 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:13.611234 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:13.611356 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:13.612308 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "a8540bd2df4d4239bddc26ea7f2cd840"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "05e74baffbee98810bba019e1672ddbb"
server_key_iv: "7c72ba9743a8cccfd14447c2afd0347f"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.612632 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8540bd2df4d4239bddc26ea7f2cd840"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "05e74baffbee98810bba019e1672ddbb"
server_key_iv: "7c72ba9743a8cccfd14447c2afd0347f"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.613757 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:13.614373 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:13.614521 5713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:13.614607 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "a8540bd2df4d4239bddc26ea7f2cd840"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "05e74baffbee98810bba019e1672ddbb"
server_key_iv: "7c72ba9743a8cccfd14447c2afd0347f"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.614672 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:13.631102 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:13.631433 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:13.631577 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:13.635396 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44427
I20260924 05:04:13.635437 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44427 every 8 connection(s)
I20260924 05:04:13.635772 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:13.636420 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:13.636605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5713
I20260924 05:04:13.636698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:13.636859 392 external_mini_cluster.cc:1428] Setting key 2fcd6185d1c4b2ab21902bb43c58f791
I20260924 05:04:13.639891 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840: Bootstrap starting.
I20260924 05:04:13.640482 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:13.640723 5780 log.cc:824] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:13.641326 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840: No bootstrap required, opened a new log
I20260924 05:04:13.642393 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.637344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.62:44427)
0924 05:04:13.637534 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.637543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.637576 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:04:13.638116 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.638126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.638211 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.638371 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.638521 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.639860 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.640635 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.641063 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.641174 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.641826 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.641831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.641831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.641988 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.642018 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.642028 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.642083 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.642163 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.642240 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260924 05:04:13.643251 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:04:13.643289 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44427 } }
I20260924 05:04:13.643436 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:13.643479 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8540bd2df4d4239bddc26ea7f2cd840, State: Initialized, Role: FOLLOWER
I20260924 05:04:13.643584 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44427 } }
I20260924 05:04:13.643646 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:13.643667 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:13.643711 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.644318 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44427 } }
I20260924 05:04:13.644404 5780 leader_election.cc:302] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8540bd2df4d4239bddc26ea7f2cd840; no voters:
I20260924 05:04:13.644546 5780 leader_election.cc:288] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:13.644590 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:13.644726 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [term 1 LEADER]: Becoming Leader. State: Replica: a8540bd2df4d4239bddc26ea7f2cd840, State: Running, Role: LEADER
I20260924 05:04:13.644779 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:13.644819 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44427 } }
I20260924 05:04:13.645148 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44427 } } }
I20260924 05:04:13.645153 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8540bd2df4d4239bddc26ea7f2cd840. Latest consensus state: current_term: 1 leader_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8540bd2df4d4239bddc26ea7f2cd840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44427 } } }
I20260924 05:04:13.645247 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.645242 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.645447 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:13.645928 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:13.647213 5794 catalog_manager.cc:1347] Generated new cluster ID: a5192ac728304706aacc5e16e76134c4
I20260924 05:04:13.647262 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:13.652330 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:13.652782 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:13.659008 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8540bd2df4d4239bddc26ea7f2cd840: Generated new TSK 0
I20260924 05:04:13.659147 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:13.667171 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.663500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.0.98.62:44427)
0924 05:04:13.663550 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.663551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.663649 (+ 98us) server_negotiation.cc:407] Connection header received
0924 05:04:13.663728 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.663728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.663738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.663788 (+ 50us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:04:13.663915 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.664698 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.665823 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.666016 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.666068 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.666213 (+ 145us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:04:13.666970 (+ 757us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 05:04:13.667102 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 05:04:13.667220 5783 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 05:04:13.667250 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:13.663440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44427 (local address 127.0.0.1:56796)
0924 05:04:13.663586 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.663597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:13.663667 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:13.663803 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:13.663804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:13.663806 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:04:13.663863 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.663865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.664784 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.664785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.665750 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.665753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.666073 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.666076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.666136 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.666142 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:04:13.667058 (+ 916us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 05:04:13.667154 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260924 05:04:13.667644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5713
E20260924 05:04:13.678074 5810 webserver.cc:533] Webserver: sslize failed:
E20260924 05:04:13.678134 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T05:04:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (188 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T05:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:13Z Disabled control of system clock
I20260924 05:04:13.710605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32891
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/testchainca.cert with env {}
W20260924 05:04:13.788707 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:13.788888 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:13.788908 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:13.790042 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:13.790084 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:13.790095 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:13.790107 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:13.790120 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:04:13.790131 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:04:13.790141 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 05:04:13.791625 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:13.791802 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:13.791960 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:13.796221 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:13.796273 5814 server_base.cc:1034] running on GCE node
W20260924 05:04:13.796224 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:13.796216 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:13.796607 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:13.796859 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:13.797995 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790226253797983 us; error 28 us; skew 500 ppm
I20260924 05:04:13.799036 5814 webserver.cc:460] Webserver started at http://127.0.98.62:36013/ using document root <none> and password file <none>
I20260924 05:04:13.799247 5814 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:13.799292 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:13.799405 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:13.800338 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "131b3e6926a34b49863a861a05b23b10"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "15c54876e338e9d6da51b12883c4e379"
server_key_iv: "87d8cb284823ff65e223654e7864c028"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.800659 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "131b3e6926a34b49863a861a05b23b10"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "15c54876e338e9d6da51b12883c4e379"
server_key_iv: "87d8cb284823ff65e223654e7864c028"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.801797 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:13.802410 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:13.802556 5814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:13.802629 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "131b3e6926a34b49863a861a05b23b10"
format_stamp: "Formatted at 2026-09-24 05:04:13 on dist-test-slave-m01t"
server_key: "15c54876e338e9d6da51b12883c4e379"
server_key_iv: "87d8cb284823ff65e223654e7864c028"
server_key_version: "encryptionkey@0"
I20260924 05:04:13.802691 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:13.808964 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:13.809198 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:13.809307 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:13.812680 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32891
I20260924 05:04:13.812706 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32891 every 8 connection(s)
I20260924 05:04:13.812964 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:13.813782 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:13.814468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5814
I20260924 05:04:13.814551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:13.814706 392 external_mini_cluster.cc:1428] Setting key 3fef625cc912c3fcf07b9b02a9eec953
I20260924 05:04:13.816380 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10: Bootstrap starting.
I20260924 05:04:13.817022 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:13.817315 5881 log.cc:824] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:13.818037 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10: No bootstrap required, opened a new log
I20260924 05:04:13.819597 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.815109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.0.98.62:32891)
0924 05:04:13.815259 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.815271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.815293 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:04:13.815668 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.815674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.815728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.815832 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:13.816047 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.816902 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.817955 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.818355 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.818439 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.819038 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:13.819042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:13.819042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:13.819192 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:13.819222 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:13.819233 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:13.819278 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:13.819361 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 05:04:13.819417 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260924 05:04:13.820081 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b3e6926a34b49863a861a05b23b10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 05:04:13.820227 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:13.820259 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131b3e6926a34b49863a861a05b23b10, State: Initialized, Role: FOLLOWER
I20260924 05:04:13.820372 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b3e6926a34b49863a861a05b23b10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 05:04:13.820468 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:13.820472 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:04:13.820509 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:13.820556 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:13.821046 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b3e6926a34b49863a861a05b23b10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 05:04:13.821146 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 131b3e6926a34b49863a861a05b23b10; no voters:
I20260924 05:04:13.821290 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:13.821344 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:13.821453 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [term 1 LEADER]: Becoming Leader. State: Replica: 131b3e6926a34b49863a861a05b23b10, State: Running, Role: LEADER
I20260924 05:04:13.821528 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b3e6926a34b49863a861a05b23b10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 05:04:13.821550 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:13.822739 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 131b3e6926a34b49863a861a05b23b10. Latest consensus state: current_term: 1 leader_uuid: "131b3e6926a34b49863a861a05b23b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b3e6926a34b49863a861a05b23b10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260924 05:04:13.822822 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:13.822923 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "131b3e6926a34b49863a861a05b23b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b3e6926a34b49863a861a05b23b10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260924 05:04:13.823076 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10 [sys.catalog]: This master's current role is: LEADER
W20260924 05:04:13.823338 5901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:04:13.823376 5901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:04:13.823436 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:13.823602 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:13.825031 5902 catalog_manager.cc:1347] Generated new cluster ID: cb8436a883814f929013275d77233849
I20260924 05:04:13.825084 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:13.850158 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:13.850878 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:13.857733 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 131b3e6926a34b49863a861a05b23b10: Generated new TSK 0
I20260924 05:04:13.857906 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:13.929536 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:13.864346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.0.98.62:32891)
0924 05:04:13.864395 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:13.864397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:13.864524 (+ 127us) server_negotiation.cc:407] Connection header received
0924 05:04:13.864598 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:13.864599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:13.864610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:13.864666 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:04:13.864792 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.865290 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.866036 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.866240 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.866293 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.866457 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:04:13.929288 (+ 62831us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 05:04:13.929457 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35890: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 05:04:13.929636 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35890: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 05:04:13.929641 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:13.864309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.0.1:35890)
0924 05:04:13.864460 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:13.864469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:13.864528 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:13.864678 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:13.864679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:13.864681 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:04:13.864735 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.864738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.865359 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.865360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.865958 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:13.865960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:13.866305 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:13.866308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:13.866371 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:13.866377 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:04:13.929426 (+ 63049us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 05:04:13.929545 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 05:04:13.929947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5814
2026-09-24T05:04:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (242 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:14 dist-test-slave-m01t krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:14 dist-test-slave-m01t krb5kdc[5922](info): set up 2 sockets
Sep 24 05:04:14 dist-test-slave-m01t krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260924 05:04:15.033599 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:15.137437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42733
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:15.214744 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:15.214933 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:15.214953 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:15.216082 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:15.216123 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:15.216135 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:15.216156 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:15.216177 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:15.217634 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:15.217868 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:15.218056 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:15.222415 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.222432 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.222429 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:15.222616 5935 server_base.cc:1034] running on GCE node
I20260924 05:04:15.222858 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:15.223131 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:15.224265 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790226255224251 us; error 31 us; skew 500 ppm
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.226591 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:15.226972 5935 webserver.cc:460] Webserver started at http://127.0.98.62:44195/ using document root <none> and password file <none>
I20260924 05:04:15.227157 5935 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:15.227196 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:15.227283 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:15.228263 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "e5f8915302f24857b48a1510ef7892a2"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "1c970761cc5c51849fdb2bc26381e234"
server_key_iv: "64aa28ef4f969d11aa9d7da5decf82a7"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.228543 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5f8915302f24857b48a1510ef7892a2"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "1c970761cc5c51849fdb2bc26381e234"
server_key_iv: "64aa28ef4f969d11aa9d7da5decf82a7"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.229683 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:15.230340 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.230470 5935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:15.230547 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e5f8915302f24857b48a1510ef7892a2"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "1c970761cc5c51849fdb2bc26381e234"
server_key_iv: "64aa28ef4f969d11aa9d7da5decf82a7"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.230607 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:15.259413 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:15.261152 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:15.261287 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:15.264814 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42733
I20260924 05:04:15.264860 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42733 every 8 connection(s)
I20260924 05:04:15.265297 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:15.266016 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:15.268100 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2: Bootstrap starting.
I20260924 05:04:15.268615 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:15.268867 6003 log.cc:824] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:15.269450 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2: No bootstrap required, opened a new log
I20260924 05:04:15.270933 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8915302f24857b48a1510ef7892a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260924 05:04:15.271061 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:15.271092 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f8915302f24857b48a1510ef7892a2, State: Initialized, Role: FOLLOWER
I20260924 05:04:15.271219 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8915302f24857b48a1510ef7892a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260924 05:04:15.271282 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:15.271303 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:15.271344 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:15.271523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 5935
I20260924 05:04:15.271616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:15.271781 392 external_mini_cluster.cc:1428] Setting key 36bd2d4be6767baeb5f101e849abc81e
I20260924 05:04:15.272310 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8915302f24857b48a1510ef7892a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260924 05:04:15.272411 6003 leader_election.cc:302] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f8915302f24857b48a1510ef7892a2; no voters:
I20260924 05:04:15.272545 6003 leader_election.cc:288] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:15.272581 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:15.272703 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [term 1 LEADER]: Becoming Leader. State: Replica: e5f8915302f24857b48a1510ef7892a2, State: Running, Role: LEADER
I20260924 05:04:15.272778 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:15.272789 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8915302f24857b48a1510ef7892a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } }
I20260924 05:04:15.273205 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f8915302f24857b48a1510ef7892a2. Latest consensus state: current_term: 1 leader_uuid: "e5f8915302f24857b48a1510ef7892a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8915302f24857b48a1510ef7892a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } }
I20260924 05:04:15.273283 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:15.273430 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f8915302f24857b48a1510ef7892a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8915302f24857b48a1510ef7892a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42733 } } }
I20260924 05:04:15.273487 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:15.273859 6017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:15.274356 6017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:15.276012 6017 catalog_manager.cc:1347] Generated new cluster ID: e2789865d8d744cdb55a495bc1ede997
I20260924 05:04:15.276057 6017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:15.283113 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.272193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.98.62:42733)
0924 05:04:15.272364 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.272373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.272406 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:04:15.272520 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.272527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.272583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.272723 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.273801 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.275186 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.275982 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.276587 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.276698 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.279257 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.279261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.279261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.279272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.281610 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.281994 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.281996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.281997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.282091 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.282284 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.282285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.282285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.282563 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.282694 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.282794 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.282914 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260924 05:04:15.298877 6017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:15.299407 6017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:15.311028 6017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f8915302f24857b48a1510ef7892a2: Generated new TSK 0
I20260924 05:04:15.311199 6017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:15.340185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:15.425473 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:15.425684 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:15.425729 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:15.426908 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:15.426949 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:15.426975 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:15.428490 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:15.428725 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:15.428941 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:15.433584 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.433571 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.433573 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:15.433851 6029 server_base.cc:1034] running on GCE node
I20260924 05:04:15.434052 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:15.434274 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:15.435405 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790226255435392 us; error 27 us; skew 500 ppm
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.437661 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:15.438112 6029 webserver.cc:460] Webserver started at http://127.0.98.1:33357/ using document root <none> and password file <none>
I20260924 05:04:15.438321 6029 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:15.438381 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:15.438473 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:15.439437 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "d2031bfdcbc44cc9825739e07777b506"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "eb5fd046a3da81f2f82cc514633e32ba"
server_key_iv: "46e1dd585c7211477baa9db2e0624217"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.439721 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d2031bfdcbc44cc9825739e07777b506"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "eb5fd046a3da81f2f82cc514633e32ba"
server_key_iv: "46e1dd585c7211477baa9db2e0624217"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.440851 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:15.441499 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.441663 6029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.441768 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "d2031bfdcbc44cc9825739e07777b506"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "eb5fd046a3da81f2f82cc514633e32ba"
server_key_iv: "46e1dd585c7211477baa9db2e0624217"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.441833 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:15.457501 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:15.459415 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:15.459517 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:15.459678 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:15.459908 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:15.459949 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.459967 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:15.459983 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.465102 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45017
I20260924 05:04:15.465160 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45017 every 8 connection(s)
I20260924 05:04:15.465409 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:15.473157 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.466059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45993 (local address 127.0.98.62:42733)
0924 05:04:15.466104 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.466107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.466283 (+ 176us) server_negotiation.cc:407] Connection header received
0924 05:04:15.467132 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.467133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.467147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.467194 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.467699 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.468183 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.469059 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.469241 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.469287 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.471013 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.471014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.471015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.471021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.472197 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.472477 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.472478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.472479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.472533 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.472724 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.472725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.472725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.472816 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.472858 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.473041 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.473095 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:15.473393 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.466034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.98.1:45993)
0924 05:04:15.466174 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:15.466207 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:15.467051 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:15.467251 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:15.467254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:15.467464 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:15.467639 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:15.467643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.468239 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.468239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:15.469000 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:15.469002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.469295 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.469295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:15.469378 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.469757 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:15.469765 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:15.470935 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:15.472253 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:15.472256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:15.472257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:15.472428 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:15.472584 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:15.472585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:15.472585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:15.472680 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:15.472855 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:15.472858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:15.472953 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 05:04:15.473023 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:04:15.473122 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260924 05:04:15.473846 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42733
I20260924 05:04:15.473918 6158 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:15.474066 6158 heartbeater.cc:511] Master 127.0.98.62:42733 requested a full tablet report, sending...
I20260924 05:04:15.474501 5967 ts_manager.cc:194] Registered new tserver with Master: d2031bfdcbc44cc9825739e07777b506 (127.0.98.1:45017)
I20260924 05:04:15.474963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6029
I20260924 05:04:15.475049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:15.475232 392 external_mini_cluster.cc:1428] Setting key c175fa6c89f0abd8d206ef3e49141890
I20260924 05:04:15.475234 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:15.496469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:15.575135 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:15.575322 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:15.575342 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:15.576423 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:15.576458 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:15.576483 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:15.577883 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:15.578063 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:15.578233 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:15.582652 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:15.582782 6163 server_base.cc:1034] running on GCE node
W20260924 05:04:15.582659 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.582674 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:15.583106 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:15.583318 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:15.584458 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790226255584443 us; error 26 us; skew 500 ppm
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.586721 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:15.587097 6163 webserver.cc:460] Webserver started at http://127.0.98.2:36901/ using document root <none> and password file <none>
I20260924 05:04:15.587299 6163 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:15.587344 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:15.587463 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:15.588445 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "b3606e483afc424a90339c217a7d2187"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "3da2a7d731318275f5e972c188ea2ac7"
server_key_iv: "af4d236f6776a9bc06e5841bec14ee34"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.588761 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b3606e483afc424a90339c217a7d2187"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "3da2a7d731318275f5e972c188ea2ac7"
server_key_iv: "af4d236f6776a9bc06e5841bec14ee34"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.589973 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:15.590607 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.590756 6163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:15.590833 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "b3606e483afc424a90339c217a7d2187"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "3da2a7d731318275f5e972c188ea2ac7"
server_key_iv: "af4d236f6776a9bc06e5841bec14ee34"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.590891 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:15.599161 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:15.600685 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:15.600803 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:15.600983 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:15.601248 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:15.601279 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.601302 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:15.601326 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.606498 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35265
I20260924 05:04:15.606567 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35265 every 8 connection(s)
I20260924 05:04:15.606801 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:15.610121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6163
I20260924 05:04:15.610204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:15.610357 392 external_mini_cluster.cc:1428] Setting key 17888dfd1b1ba85fdfc358eba2c000ed
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:15.615622 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58717 (local address 127.0.98.62:42733)
0924 05:04:15.607504 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.607507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.607671 (+ 164us) server_negotiation.cc:407] Connection header received
0924 05:04:15.608560 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.608561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.608574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.608629 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.609187 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.609670 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.610791 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.611032 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.611088 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.613062 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.613063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.613064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.613071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.614435 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.614782 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.614783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.614784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.614845 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.615037 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.615038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.615039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.615178 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.615245 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.615442 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.615531 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:04:15.615664 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.607395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.98.2:58717)
0924 05:04:15.607550 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:15.607591 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:15.608461 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:15.608704 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:15.608708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:15.608936 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:15.609104 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:15.609108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.609768 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.609769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:15.610719 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:15.610722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.611101 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.611102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:15.611225 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.611663 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:15.611672 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:15.612987 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:15.614505 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:15.614510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:15.614512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:15.614711 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:15.614906 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:15.614907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:15.614907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:15.614990 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:15.615232 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:15.615235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:15.615335 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 05:04:15.615407 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:04:15.615515 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260924 05:04:15.616375 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42733
I20260924 05:04:15.616456 6292 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:15.616617 6292 heartbeater.cc:511] Master 127.0.98.62:42733 requested a full tablet report, sending...
I20260924 05:04:15.616967 5967 ts_manager.cc:194] Registered new tserver with Master: b3606e483afc424a90339c217a7d2187 (127.0.98.2:35265)
I20260924 05:04:15.617309 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58717
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:15.633498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:15.710487 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:15.710741 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:15.710772 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:15.712332 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:15.712383 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:15.712419 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:15.714403 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:15.714641 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:15.714869 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:15.720016 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.720009 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:15.720007 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:15.720234 6297 server_base.cc:1034] running on GCE node
I20260924 05:04:15.720400 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:15.720600 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:15.721755 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790226255721739 us; error 29 us; skew 500 ppm
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.724026 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:15.724435 6297 webserver.cc:460] Webserver started at http://127.0.98.3:44033/ using document root <none> and password file <none>
I20260924 05:04:15.724642 6297 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:15.724777 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:15.724924 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:15.726014 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "d38ae7885e354844a9ad5d068d5aad9f"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "7def312ef4d2435884451afa6de93eca"
server_key_iv: "4600d52b5be552e2d340bd9f8a7a204d"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.726338 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d38ae7885e354844a9ad5d068d5aad9f"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "7def312ef4d2435884451afa6de93eca"
server_key_iv: "4600d52b5be552e2d340bd9f8a7a204d"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.727523 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:15.728122 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.728274 6297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:15.728350 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "d38ae7885e354844a9ad5d068d5aad9f"
format_stamp: "Formatted at 2026-09-24 05:04:15 on dist-test-slave-m01t"
server_key: "7def312ef4d2435884451afa6de93eca"
server_key_iv: "4600d52b5be552e2d340bd9f8a7a204d"
server_key_version: "encryptionkey@0"
I20260924 05:04:15.728407 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:15.755347 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:15.757431 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:15.757563 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:15.757827 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:15.758121 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:15.758154 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.758176 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:15.758199 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:15.763258 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38035
I20260924 05:04:15.763298 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38035 every 8 connection(s)
I20260924 05:04:15.763566 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:15.768000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6297
I20260924 05:04:15.768095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:15.768244 392 external_mini_cluster.cc:1428] Setting key 57c51b04def86972ae6f30d047c314e0
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:15.771798 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.764191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34661 (local address 127.0.98.62:42733)
0924 05:04:15.764248 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.764249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.764406 (+ 157us) server_negotiation.cc:407] Connection header received
0924 05:04:15.765282 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.765283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.765292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.765331 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.765914 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.766386 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.767262 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.767432 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.767474 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.769439 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.769440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.769440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.769447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.770655 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.770985 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.770986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.770987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.771038 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.771237 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.771238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.771238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.771342 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.771394 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.771643 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.771714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:04:15.771945 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.764150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42733 (local address 127.0.98.3:34661)
0924 05:04:15.764294 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:15.764333 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:15.765191 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:15.765386 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:15.765389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:15.765621 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:15.765847 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:15.765851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.766443 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.766443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:15.767208 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:15.767210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.767482 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.767482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:15.767566 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.767901 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:15.767908 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:15.769359 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:15.770724 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:15.770729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:15.770731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:15.770923 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:15.771087 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:15.771088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:15.771088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:15.771185 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:15.771393 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:15.771399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:15.771529 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 05:04:15.771613 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:04:15.771749 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260924 05:04:15.772356 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42733
I20260924 05:04:15.772437 6426 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:15.772594 6426 heartbeater.cc:511] Master 127.0.98.62:42733 requested a full tablet report, sending...
I20260924 05:04:15.772918 5967 ts_manager.cc:194] Registered new tserver with Master: d38ae7885e354844a9ad5d068d5aad9f (127.0.98.3:38035)
I20260924 05:04:15.773336 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34661
I20260924 05:04:15.779758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/test-admin.keytab.
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.793591 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 05:04:15.795051 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:15.794482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.0.98.1:45017)
0924 05:04:15.794623 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.794628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.794639 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:15.794663 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.794665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.794678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.794738 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.794994 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39900: BlockingRecv error: recv got EOF from 127.0.0.1:39900 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
W20260924 05:04:15.795168 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.796036 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.796762 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:04:15.803520 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.797207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.0.98.1:45017)
0924 05:04:15.797255 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.797256 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.797301 (+ 45us) server_negotiation.cc:407] Connection header received
0924 05:04:15.797357 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.797358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.797369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.797416 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.797781 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.798484 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.799156 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.799471 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.799528 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.801261 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.801269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.801269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.801276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.802524 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.802812 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.802815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.802816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.802865 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.803032 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.803033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.803034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.803156 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.803282 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.803359 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.803433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 05:04:15.804886 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:15.804492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.98.1:45017)
0924 05:04:15.804538 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.804539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.804546 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:15.804597 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.804598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.804606 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.804650 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.804852 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39908: BlockingRecv error: recv got EOF from 127.0.0.1:39908 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 05:04:15.804958 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.805748 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.806457 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:04:15.811980 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.806839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.0.98.1:45017)
0924 05:04:15.806871 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.806873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.806882 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:15.806925 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.806926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.806933 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.806970 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.807229 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.807666 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.808262 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.808440 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.808478 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.810054 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.810055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.810055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.810060 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.811233 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.811504 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.811505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.811505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.811548 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.811667 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.811668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.811668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.811737 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.811791 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.811860 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.811910 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:04:15.813249 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:15.812758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.0.98.1:45017)
0924 05:04:15.812809 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.812810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.812859 (+ 49us) server_negotiation.cc:407] Connection header received
0924 05:04:15.812913 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.812914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.812922 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.812964 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.813210 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39928: BlockingRecv error: recv got EOF from 127.0.0.1:39928 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 05:04:15.813299 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226228698807-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.814098 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:15.814805 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:04:15.820181 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:15.815186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.98.1:45017)
0924 05:04:15.815218 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:15.815220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:15.815247 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:04:15.815288 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:15.815289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:15.815295 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:15.815332 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:15.815601 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.816029 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.816604 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:15.816762 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:15.816800 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:15.818466 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:15.818467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:15.818467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:15.818472 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:15.819421 (+ 949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.819705 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.819705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.819706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.819744 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:15.819865 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:15.819865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:15.819866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:15.819941 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:15.819996 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:15.820066 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 05:04:15.820110 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:15.820665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6029
I20260924 05:04:15.824962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6163
I20260924 05:04:15.828719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6297
I20260924 05:04:15.832247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 5935
2026-09-24T05:04:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1736 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[6446](info): set up 2 sockets
Sep 24 05:04:15 dist-test-slave-m01t krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260924 05:04:16.777572 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:16 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 05:04:16.879918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:16.956789 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:16.956969 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:16.956990 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:16.958117 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:16.958153 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:16.958164 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:16.958175 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:16.958192 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:16.959568 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:16.959744 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:16.959897 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:16.964072 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:16.964145 6459 server_base.cc:1034] running on GCE node
W20260924 05:04:16.964077 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:16.964071 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:16.964509 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:16.964756 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:16.965873 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790226256965863 us; error 26 us; skew 500 ppm
Sep 24 05:04:16 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:16.970786 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 05:04:16.971184 6459 webserver.cc:460] Webserver started at http://127.0.98.62:43191/ using document root <none> and password file <none>
I20260924 05:04:16.971361 6459 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:16.971405 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:16.971521 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:16.972458 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "1a98c13313924495b97777015844226a"
format_stamp: "Formatted at 2026-09-24 05:04:16 on dist-test-slave-m01t"
server_key: "30e66aa215aabb42ba330a116c8ca21a"
server_key_iv: "34839384426abbcd97cbb03e6d486498"
server_key_version: "encryptionkey@0"
I20260924 05:04:16.972759 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a98c13313924495b97777015844226a"
format_stamp: "Formatted at 2026-09-24 05:04:16 on dist-test-slave-m01t"
server_key: "30e66aa215aabb42ba330a116c8ca21a"
server_key_iv: "34839384426abbcd97cbb03e6d486498"
server_key_version: "encryptionkey@0"
I20260924 05:04:16.973984 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:16.974577 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:16.974714 6459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:16.974788 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "1a98c13313924495b97777015844226a"
format_stamp: "Formatted at 2026-09-24 05:04:16 on dist-test-slave-m01t"
server_key: "30e66aa215aabb42ba330a116c8ca21a"
server_key_iv: "34839384426abbcd97cbb03e6d486498"
server_key_version: "encryptionkey@0"
I20260924 05:04:16.974843 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:16.984381 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:16.985852 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:16.985967 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:16.989398 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42581
I20260924 05:04:16.989425 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42581 every 8 connection(s)
I20260924 05:04:16.989706 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:16.990547 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:16.992552 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Bootstrap starting.
I20260924 05:04:16.993049 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:16.993283 6527 log.cc:824] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:16.993492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6459
I20260924 05:04:16.993583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:16.993793 392 external_mini_cluster.cc:1428] Setting key 1acc40883f8091689019203b46a68830
I20260924 05:04:16.993965 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: No bootstrap required, opened a new log
I20260924 05:04:16.995687 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } }
I20260924 05:04:16.995810 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:16.995842 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a98c13313924495b97777015844226a, State: Initialized, Role: FOLLOWER
I20260924 05:04:16.995955 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } }
I20260924 05:04:16.996024 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:16.996053 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:16.996097 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:16.996608 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } }
I20260924 05:04:16.996716 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a98c13313924495b97777015844226a; no voters:
I20260924 05:04:16.996861 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:16.996891 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:16.996985 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 LEADER]: Becoming Leader. State: Replica: 1a98c13313924495b97777015844226a, State: Running, Role: LEADER
I20260924 05:04:16.997059 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } }
I20260924 05:04:16.997092 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:16.997345 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a98c13313924495b97777015844226a. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } }
I20260924 05:04:16.997417 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:16.997354 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } }
I20260924 05:04:16.997459 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:16.998171 6537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:16.998631 6537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:04:16 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:17.001641 6537 catalog_manager.cc:1347] Generated new cluster ID: dde3d3ee0e5f402eaa284404d019683e
I20260924 05:04:17.001685 6537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:17.004400 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:16.994236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.98.62:42581)
0924 05:04:16.994399 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:16.994408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:16.994438 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:04:16.995076 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:16.995083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:16.995133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:16.995251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:16.995829 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:16.997348 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:16.998363 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:16.998926 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:16.999021 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.001091 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.001095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.001095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.001106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.002935 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.003319 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.003321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.003321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.003410 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.003617 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.003617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.003618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.003869 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.004004 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.004099 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.004220 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260924 05:04:17.017884 6537 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:17.018419 6537 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:17.022619 6537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Generated new TSK 0
I20260924 05:04:17.022794 6537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 05:04:17.049249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42581
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:17.126669 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:17.126873 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:17.126896 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:17.128017 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:17.128060 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:17.128085 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:17.129456 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:17.129647 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:17.129877 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.134527 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.134621 6553 server_base.cc:1034] running on GCE node
W20260924 05:04:17.134523 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.134522 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.134914 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.135144 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.136278 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790226257136267 us; error 27 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.138686 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:04:17.139074 6553 webserver.cc:460] Webserver started at http://127.0.98.1:43935/ using document root <none> and password file <none>
I20260924 05:04:17.139285 6553 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.139333 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.139453 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:17.140450 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "87e7066d8cc64d23b9e155c02ae63c82"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "18a9c37e396d62c41be2934323e5a210"
server_key_iv: "53dbea89ca93bafbb64e44f4768f453b"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.140762 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "87e7066d8cc64d23b9e155c02ae63c82"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "18a9c37e396d62c41be2934323e5a210"
server_key_iv: "53dbea89ca93bafbb64e44f4768f453b"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.141948 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:17.142565 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.142705 6553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:17.142781 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "87e7066d8cc64d23b9e155c02ae63c82"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "18a9c37e396d62c41be2934323e5a210"
server_key_iv: "53dbea89ca93bafbb64e44f4768f453b"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.142836 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.149160 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.151072 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:17.151196 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.151368 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:17.151639 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:17.151671 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.151696 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:17.151719 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.156804 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40019
I20260924 05:04:17.156853 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40019 every 8 connection(s)
I20260924 05:04:17.157096 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:17.163378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6553
I20260924 05:04:17.163461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:17.163632 392 external_mini_cluster.cc:1428] Setting key 3283e954134748ee31c8b96909cf883a
I20260924 05:04:17.165031 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.157688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60817 (local address 127.0.98.62:42581)
0924 05:04:17.157756 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.157758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.157918 (+ 160us) server_negotiation.cc:407] Connection header received
0924 05:04:17.158341 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.158342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.158352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.158399 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.159204 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.159697 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.160587 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.160782 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.160822 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.162602 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.162603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.162604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.162609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.163851 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.164212 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.164213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.164213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.164271 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.164510 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.164511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.164511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.164599 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.164647 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.164911 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.164971 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:04:17.165246 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.157663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:60817)
0924 05:04:17.157815 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.157846 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.158273 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.158448 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.158452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.158954 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.159139 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.159143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.159755 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.159756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.160530 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.160532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.160832 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.160833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.160924 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.161277 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.161284 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.162518 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.163905 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.163911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.163913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.164139 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.164327 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.164328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.164328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.164456 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.164648 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.164652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.164796 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.164891 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.165047 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260924 05:04:17.165939 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42581
I20260924 05:04:17.166033 6682 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:17.166200 6682 heartbeater.cc:511] Master 127.0.98.62:42581 requested a full tablet report, sending...
I20260924 05:04:17.166625 6491 ts_manager.cc:194] Registered new tserver with Master: 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019)
I20260924 05:04:17.167179 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60817
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 05:04:17.184998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42581
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:17.262728 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:17.262907 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:17.262935 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:17.263993 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:17.264029 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:17.264061 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:17.265398 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:17.265599 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:17.265821 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.270231 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.270226 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.270210 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.270465 6687 server_base.cc:1034] running on GCE node
I20260924 05:04:17.270637 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.270840 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.271973 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790226257271959 us; error 27 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.274252 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:04:17.274647 6687 webserver.cc:460] Webserver started at http://127.0.98.2:42937/ using document root <none> and password file <none>
I20260924 05:04:17.274819 6687 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.274864 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.274981 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:17.275884 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "344cc91000b8475bad832b85c56a1bb7"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "2287a27587f9babda96e00c31d9dc508"
server_key_iv: "2e0c23fb17e1963665a4a1d35ee43751"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.276197 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "344cc91000b8475bad832b85c56a1bb7"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "2287a27587f9babda96e00c31d9dc508"
server_key_iv: "2e0c23fb17e1963665a4a1d35ee43751"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.277271 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:17.277877 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.278012 6687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.278084 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "344cc91000b8475bad832b85c56a1bb7"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "2287a27587f9babda96e00c31d9dc508"
server_key_iv: "2e0c23fb17e1963665a4a1d35ee43751"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.278162 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.294019 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.296036 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:17.296170 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.296366 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:17.296653 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:17.296694 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.296720 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:17.296742 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.302084 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36027
I20260924 05:04:17.302120 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36027 every 8 connection(s)
I20260924 05:04:17.302407 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:17.308550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6687
I20260924 05:04:17.308634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:17.308825 392 external_mini_cluster.cc:1428] Setting key 08ad885fadd3909783442ae937b7ef22
I20260924 05:04:17.310458 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.302983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59343 (local address 127.0.98.62:42581)
0924 05:04:17.303055 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.303057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.303219 (+ 162us) server_negotiation.cc:407] Connection header received
0924 05:04:17.303673 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.303674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.303684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.303731 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.304543 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.305018 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.305916 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.306092 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.306141 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.307861 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.307862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.307862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.307868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.309217 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.309610 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.309611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.309611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.309667 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.309895 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.309896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.309896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.310005 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.310069 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.310300 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.310373 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:04:17.310623 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.302961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:59343)
0924 05:04:17.303099 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.303133 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.303598 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.303783 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.303786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.304294 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.304478 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.304483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.305075 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.305075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.305857 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.305859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.306146 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.306147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.306234 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.306575 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.306582 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.307789 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.309282 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.309288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.309290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.309545 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.309737 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.309738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.309738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.309841 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.310059 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.310065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.310191 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.310275 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.310413 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260924 05:04:17.311123 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42581
I20260924 05:04:17.311197 6816 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:17.311349 6816 heartbeater.cc:511] Master 127.0.98.62:42581 requested a full tablet report, sending...
I20260924 05:04:17.311671 6491 ts_manager.cc:194] Registered new tserver with Master: 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:17.312006 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59343
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 05:04:17.329927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42581
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:17.407253 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:17.407429 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:17.407459 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:17.408533 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:17.408571 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:17.408603 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:17.409992 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:17.410214 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:17.410401 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.414741 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.414732 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.414734 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.415031 6821 server_base.cc:1034] running on GCE node
I20260924 05:04:17.415220 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.415436 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.416567 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790226257416555 us; error 26 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.418767 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:04:17.419157 6821 webserver.cc:460] Webserver started at http://127.0.98.3:46285/ using document root <none> and password file <none>
I20260924 05:04:17.419348 6821 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.419392 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.419508 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:17.420454 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "9cf97fa9703e45819ac3330ab207cac5"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "a8dc1bacc9739b58ed548089f1900f45"
server_key_iv: "b707e1db7d080dd103a5d3b0526b884c"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.420756 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9cf97fa9703e45819ac3330ab207cac5"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "a8dc1bacc9739b58ed548089f1900f45"
server_key_iv: "b707e1db7d080dd103a5d3b0526b884c"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.421928 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:17.422549 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.422688 6821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.422762 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "9cf97fa9703e45819ac3330ab207cac5"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "a8dc1bacc9739b58ed548089f1900f45"
server_key_iv: "b707e1db7d080dd103a5d3b0526b884c"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.422819 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.434563 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.436609 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:17.436740 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.436949 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:17.437224 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:17.437256 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.437284 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:17.437307 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.442615 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36241
I20260924 05:04:17.442672 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36241 every 8 connection(s)
I20260924 05:04:17.442942 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:17.443760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6821
I20260924 05:04:17.443849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:17.444010 392 external_mini_cluster.cc:1428] Setting key 82f63186e359b172c77eaaa3dbba256f
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:17.451176 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.443498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45015 (local address 127.0.98.62:42581)
0924 05:04:17.443538 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.443540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.443730 (+ 190us) server_negotiation.cc:407] Connection header received
0924 05:04:17.444323 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.444324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.444336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.444385 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.445283 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.445857 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.446892 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.447103 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.447158 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.448953 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.448955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.448955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.448962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.450012 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.450352 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.450353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.450353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.450401 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.450611 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.450612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.450612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.450714 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.450769 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.451009 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.451089 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:17.451321 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.443473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:45015)
0924 05:04:17.443615 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.443651 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.444228 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.444465 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.444469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.445007 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.445195 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.445200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.445932 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.445933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.446814 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.446816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.447168 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.447169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.447294 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.447673 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.447681 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.448869 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.450087 (+ 1218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.450092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.450094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.450288 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.450452 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.450453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.450453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.450556 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.450768 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.450774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.450898 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.450981 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.451119 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260924 05:04:17.451706 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42581
I20260924 05:04:17.451786 6950 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:17.451920 6950 heartbeater.cc:511] Master 127.0.98.62:42581 requested a full tablet report, sending...
I20260924 05:04:17.452236 6491 ts_manager.cc:194] Registered new tserver with Master: 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241)
I20260924 05:04:17.452569 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45015
I20260924 05:04:17.455369 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 05:04:17.476507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45087
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42581,127.0.98.61:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:17.554019 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:17.554209 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:17.554237 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:17.555315 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:17.555352 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:17.555370 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:17.555385 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:17.555401 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:17.556787 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42581,127.0.98.61:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
W20260924 05:04:17.556886 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42581,127.0.98.61:45087'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 05:04:17.557039 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:17.557215 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.561414 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.561421 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.561596 6955 server_base.cc:1034] running on GCE node
W20260924 05:04:17.561455 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.561870 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.562127 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.563254 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790226257563238 us; error 31 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.565328 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 05:04:17.565750 6955 webserver.cc:460] Webserver started at http://127.0.98.61:40589/ using document root <none> and password file <none>
I20260924 05:04:17.565937 6955 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.565989 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.566107 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:17.567047 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/instance:
uuid: "c18a88efbe12492d93b70836c35441e0"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "19244f6d725f0e411604f3ac7662d9be"
server_key_iv: "34a2fe9e0c850136cbc69542e533d93a"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.567353 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal/instance:
uuid: "c18a88efbe12492d93b70836c35441e0"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "19244f6d725f0e411604f3ac7662d9be"
server_key_iv: "34a2fe9e0c850136cbc69542e533d93a"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.568466 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:17.569049 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.569192 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:17.569265 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
uuid: "c18a88efbe12492d93b70836c35441e0"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "19244f6d725f0e411604f3ac7662d9be"
server_key_iv: "34a2fe9e0c850136cbc69542e533d93a"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.569320 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.579406 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.580927 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:17.581142 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.584983 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45087
I20260924 05:04:17.585006 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45087 every 8 connection(s)
I20260924 05:04:17.585296 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:04:17.586155 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:17.587459 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:04:17.590086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 6955
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:04:17.595968 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.588053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.0.98.62:42581)
0924 05:04:17.588110 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.588112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.588271 (+ 159us) server_negotiation.cc:407] Connection header received
0924 05:04:17.588702 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.588703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.588712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.588759 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.589247 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.589755 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.590962 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.591157 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.591207 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.593254 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.593255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.593255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.593262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.594677 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.595066 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.595067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.595067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.595116 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.595351 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.595351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.595352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.595475 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.595532 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.595808 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.595878 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:04:17.596118 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.588035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37192)
0924 05:04:17.588173 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.588203 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.588631 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.588808 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.588811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.589017 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.589190 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.589193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.589815 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.589816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.590889 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.590891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.591225 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.591226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.591355 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.591787 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.591795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.593185 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.594747 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.594753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.594754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.594994 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.595172 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.595173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.595173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.595295 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.595524 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.595530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.595682 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.595794 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.595933 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260924 05:04:17.596693 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:04:17.598667 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.590506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.98.61:45087)
0924 05:04:17.590650 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.590653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.590667 (+ 14us) server_negotiation.cc:407] Connection header received
0924 05:04:17.590684 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.590685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.590699 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.590757 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.591333 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.591990 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.592753 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.593181 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.593250 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.595910 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.595916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.595917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.595922 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.597641 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.597963 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.597966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.597966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.598018 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.598191 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.598192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.598192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.598338 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.598462 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.598549 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.598599 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:04:17.602842 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.596903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39804)
0924 05:04:17.596934 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.596943 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.596975 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.597129 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.597130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.597295 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.597341 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.597343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.598073 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.598074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.598696 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.598698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.599188 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.599188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.599230 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.599473 (+ 243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.599476 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.600386 (+ 910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.602171 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.602172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.602173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.602318 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.602457 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.602457 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.602457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.602498 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.602641 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.602641 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.602670 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.602712 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.602772 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 05:04:17.602919 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.596932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.0.98.61:45087)
0924 05:04:17.597045 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.597047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.597053 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:17.597065 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.597066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.597072 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.597120 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.597396 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.598010 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.598774 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.599095 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.599151 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.600459 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.600460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.600460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.600465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.602106 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.602374 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.602375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.602375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.602414 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.602529 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.602529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.602529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.602601 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.602645 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.602726 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.602811 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260924 05:04:17.603935 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Bootstrap starting.
I20260924 05:04:17.604446 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:17.604650 7023 log.cc:824] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:17.605190 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: No bootstrap required, opened a new log
I20260924 05:04:17.606459 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } }
I20260924 05:04:17.606565 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:17.606585 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18a88efbe12492d93b70836c35441e0, State: Initialized, Role: FOLLOWER
I20260924 05:04:17.606673 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } }
I20260924 05:04:17.606899 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:17.606868 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } } }
I20260924 05:04:17.606954 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:04:17.608237 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:04:17.608289 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:04:17.609535 6955 master_runner.cc:428] Detected that this master c18a88efbe12492d93b70836c35441e0 is joining an existing cluster
I20260924 05:04:17.609572 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45087
I20260924 05:04:17.616223 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.610689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39806)
0924 05:04:17.610797 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.610807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.610854 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.610969 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.610969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.611103 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.611151 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.611152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.611753 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.611754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.612579 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.612581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.612937 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.612937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.612969 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.613486 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.613489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.613867 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.615256 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.615257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.615399 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.615555 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.615556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.615600 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.615899 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.615900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.615931 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.616008 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.616100 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260924 05:04:17.616285 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.610631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.0.98.61:45087)
0924 05:04:17.610661 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.610663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.610845 (+ 182us) server_negotiation.cc:407] Connection header received
0924 05:04:17.610895 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.610896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.610904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.610962 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.611211 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.611690 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.612659 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.612858 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.612884 (+ 26us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.613925 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.613925 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.613926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.613931 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.615188 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615452 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.615453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.615453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.615497 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615672 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.615674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.615674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.615813 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.615878 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.616033 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.616163 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":16}
I20260924 05:04:17.616658 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.610412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.0.98.62:42581)
0924 05:04:17.610444 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.610446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.610583 (+ 137us) server_negotiation.cc:407] Connection header received
0924 05:04:17.610633 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.610634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.610643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.610692 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.611012 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.611732 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.612671 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.612958 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.613021 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.613866 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.613867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.613868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.613874 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.615447 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615768 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.615769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.615769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.615820 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615980 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.615980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.615981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.616093 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.616147 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.616515 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.616578 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:17.616665 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.610376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37202)
0924 05:04:17.610525 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.610534 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.610583 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.610698 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.610698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.610856 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.610924 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.610925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.611805 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.611806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.612583 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.612584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.613028 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.613028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.613090 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.613377 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.613381 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.613796 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.615523 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615523 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.615523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.615711 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.615871 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.615872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.615872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.615932 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.616172 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.616173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.616224 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.616471 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.616559 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:04:17.624464 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.618552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.98.61:45087)
0924 05:04:17.618581 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.618582 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.618718 (+ 136us) server_negotiation.cc:407] Connection header received
0924 05:04:17.618785 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.618786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.618794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.618836 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.619189 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.619606 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.620299 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.620495 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.620544 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.622374 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.622375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.622375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.622382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.623487 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.623803 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.623804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.623804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.623849 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.624002 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.624003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.624003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.624094 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.624144 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.624306 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.624374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 05:04:17.624486 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.618517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39822)
0924 05:04:17.618635 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.618668 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.618729 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.618839 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.618842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.619082 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.619121 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.619123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.619670 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.619671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.620231 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.620232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.620551 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.620552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.620634 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.621020 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.621024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.622301 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.623555 (+ 1254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.623562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.623564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.623743 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.623899 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.623900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.623900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.623955 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.624152 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.624158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.624208 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.624276 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.624383 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260924 05:04:17.625167 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45087: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c18a88efbe12492d93b70836c35441e0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45087
}
attrs {
promote: true
}
}
dest_uuid: "1a98c13313924495b97777015844226a"
cas_config_opid_index: -1
I20260924 05:04:17.625468 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } }
I20260924 05:04:17.625926 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c18a88efbe12492d93b70836c35441e0 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } }
I20260924 05:04:17.626236 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:17.626997 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45087
I20260924 05:04:17.627012 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Refusing update from remote peer 1a98c13313924495b97777015844226a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 05:04:17.627210 6955 master.cc:537] Master@127.0.98.61:45087 shutting down...
I20260924 05:04:17.628018 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:17.629114 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } }
I20260924 05:04:17.629189 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
W20260924 05:04:17.629765 6478 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 05:04:17.629806 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:17.629890 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:17.629930 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: stopping tablet replica
W20260924 05:04:17.630133 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a -> Peer c18a88efbe12492d93b70836c35441e0 (127.0.98.61:45087): Couldn't send request to peer c18a88efbe12492d93b70836c35441e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 05:04:17.642688 6955 master.cc:552] Master@127.0.98.61:45087 shutdown complete.
I20260924 05:04:17.642736 6955 master_runner.cc:305] Clearing existing system tablet
I20260924 05:04:17.642844 6955 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.642885 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.643438 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:17.643867 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.644011 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.644065 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.644089 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
uuid: "c18a88efbe12492d93b70836c35441e0"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "19244f6d725f0e411604f3ac7662d9be"
server_key_iv: "34a2fe9e0c850136cbc69542e533d93a"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.644765 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:17.645994 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 05:04:17.646049 6955 log.cc:1196] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 05:04:17.646201 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Deleting consensus metadata
I20260924 05:04:17.646301 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:42581
I20260924 05:04:17.646790 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42581
I20260924 05:04:17.652220 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.647125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.0.98.62:42581)
0924 05:04:17.647166 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.647168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.647284 (+ 116us) server_negotiation.cc:407] Connection header received
0924 05:04:17.647346 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.647347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.647357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.647408 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.647691 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.648176 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.648870 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.649064 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.649116 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.650075 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.650077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.650077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.650084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.651289 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.651570 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.651571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.651572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.651620 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.651778 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.651779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.651779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.651881 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.651935 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.652063 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.652127 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:17.652236 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.647070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37204)
0924 05:04:17.647214 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.647223 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.647288 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.647414 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.647415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.647570 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.647621 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.647623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.648242 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.648243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.648797 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.648798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.649119 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.649127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.649189 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.649660 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.649665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.650000 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.651357 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.651358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.651358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.651513 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.651677 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.651677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.651678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.651730 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.651933 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.651934 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.651976 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.652035 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.652137 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260924 05:04:17.652661 6511 tablet_copy_service.cc:140] P 1a98c13313924495b97777015844226a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c18a88efbe12492d93b70836c35441e0 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37204)
I20260924 05:04:17.652733 6511 tablet_copy_service.cc:161] P 1a98c13313924495b97777015844226a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c18a88efbe12492d93b70836c35441e0 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37204: session id = c18a88efbe12492d93b70836c35441e0-00000000000000000000000000000000
I20260924 05:04:17.653292 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 05:04:17.653861 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 05:04:17.654618 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: tablet copy: Starting download of 0 data blocks...
I20260924 05:04:17.654757 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: tablet copy: Starting download of 1 WAL segments...
I20260924 05:04:17.655450 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 05:04:17.656121 6511 tablet_copy_service.cc:342] P 1a98c13313924495b97777015844226a: Request end of tablet copy session c18a88efbe12492d93b70836c35441e0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37204
I20260924 05:04:17.656167 6511 tablet_copy_service.cc:433] P 1a98c13313924495b97777015844226a: ending tablet copy session c18a88efbe12492d93b70836c35441e0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c18a88efbe12492d93b70836c35441e0
I20260924 05:04:17.658324 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.659301 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.659319 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.659327 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.659461 6955 server_base.cc:1034] running on GCE node
I20260924 05:04:17.659614 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.659715 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.660820 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790226257660812 us; error 20 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.661973 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 05:04:17.662197 6955 webserver.cc:460] Webserver started at http://127.0.98.61:45627/ using document root <none> and password file <none>
I20260924 05:04:17.662302 6955 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.662334 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.662853 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:17.663295 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.663403 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.663442 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
uuid: "c18a88efbe12492d93b70836c35441e0"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "19244f6d725f0e411604f3ac7662d9be"
server_key_iv: "34a2fe9e0c850136cbc69542e533d93a"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.663491 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.684684 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.686085 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.688695 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45087
I20260924 05:04:17.688711 7126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45087 every 8 connection(s)
I20260924 05:04:17.688982 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:04:17.689860 7127 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:04:17.690382 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Bootstrap starting.
I20260924 05:04:17.693148 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:04:17.693348 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: Bootstrap complete.
I20260924 05:04:17.693528 7127 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } }
I20260924 05:04:17.693621 7127 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c18a88efbe12492d93b70836c35441e0, State: Initialized, Role: LEARNER
I20260924 05:04:17.693657 7127 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } }
I20260924 05:04:17.693854 7127 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:17.693846 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } }
I20260924 05:04:17.693943 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: LEARNER
I20260924 05:04:17.695318 7141 catalog_manager.cc:1259] Loaded cluster ID: dde3d3ee0e5f402eaa284404d019683e
I20260924 05:04:17.695362 7141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: loading cluster ID for follower catalog manager: success
I20260924 05:04:17.696336 7141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: acquiring CA information for follower catalog manager: success
I20260924 05:04:17.696578 7141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:04:17.706429 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.700142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.0.98.61:45087)
0924 05:04:17.700240 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.700241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.700308 (+ 67us) server_negotiation.cc:407] Connection header received
0924 05:04:17.700378 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.700378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.700386 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.700445 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.701055 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.701657 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.702611 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.702785 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.702827 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.704089 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.704090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.704090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.704096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.705296 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.705651 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.705652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.705652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.705728 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.705887 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.705888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.705888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.705974 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.706038 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.706146 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.706196 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 05:04:17.729580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39645
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:17.806308 7145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:17.806486 7145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:17.806505 7145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:17.807582 7145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:17.807626 7145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:17.807649 7145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:17.807672 7145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:17.807693 7145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:17.809105 7145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:17.809315 7145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:17.809501 7145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.813989 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.814072 7150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.814170 7145 server_base.cc:1034] running on GCE node
W20260924 05:04:17.814013 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.814401 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.814634 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.815762 7145 hybrid_clock.cc:648] HybridClock initialized: now 1790226257815749 us; error 26 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.817993 7145 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 05:04:17.818341 7145 webserver.cc:460] Webserver started at http://127.0.98.60:40173/ using document root <none> and password file <none>
I20260924 05:04:17.818502 7145 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.818539 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.818622 7145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:17.819512 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/instance:
uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "8ca6740f507118451afaf0327f3911b9"
server_key_iv: "d187050c15bb2590592e07d665285bcf"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.819782 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal/instance:
uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "8ca6740f507118451afaf0327f3911b9"
server_key_iv: "d187050c15bb2590592e07d665285bcf"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.820890 7145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:17.821478 7160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.821614 7145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:17.821689 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "8ca6740f507118451afaf0327f3911b9"
server_key_iv: "d187050c15bb2590592e07d665285bcf"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.821772 7145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.834995 7145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.836665 7145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:17.836772 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.840293 7145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39645
I20260924 05:04:17.840337 7212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39645 every 8 connection(s)
I20260924 05:04:17.840657 7145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:04:17.841516 7213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:17.842988 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:04:17.843742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 7145
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:04:17.851395 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.843733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.98.62:42581)
0924 05:04:17.843779 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.843781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.843907 (+ 126us) server_negotiation.cc:407] Connection header received
0924 05:04:17.844432 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.844434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.844444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.844486 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.845058 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.845544 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.846651 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.846902 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.846950 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.848998 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.848999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.849000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.849006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.850236 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.850580 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.850581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.850582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.850632 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.850838 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.850839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.850839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.850938 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.850998 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.851244 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.851306 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:04:17.851572 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.843609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37206)
0924 05:04:17.843758 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.843838 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.844328 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.844516 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.844519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.844787 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.844929 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.844933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.845629 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.845630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.846581 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.846583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.846961 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.846962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.847083 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.847555 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.847564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.848924 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.850304 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.850309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.850311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.850521 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.850683 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.850684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.850684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.850785 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.851007 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.851012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.851130 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.851222 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.851360 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260924 05:04:17.852105 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.844186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.0.98.60:39645)
0924 05:04:17.844313 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.844317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.844347 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:04:17.844426 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.844428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.844447 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.844527 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.845065 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.846062 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.846804 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.847136 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.847200 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.849489 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.849494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.849495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.849500 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.850913 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.851198 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.851201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.851201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.851258 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.851523 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.851524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.851524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.851669 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.851790 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.851944 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.852015 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260924 05:04:17.852180 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:04:17.857179 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.852375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39850)
0924 05:04:17.852413 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.852421 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.852458 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.852631 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.852632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.852736 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.852774 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.852775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.853319 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.853319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.853825 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.853826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.854116 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.854117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.854159 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.854469 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.854474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.855430 (+ 956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.856548 (+ 1118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.856549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.856549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.856685 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.856815 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.856815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.856815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.856855 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.856999 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.856999 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.857027 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.857072 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.857128 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:04:17.857199 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.852477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.0.98.61:45087)
0924 05:04:17.852519 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.852521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.852528 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:17.852551 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.852552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.852563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.852619 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.852827 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.853270 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.853880 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.854066 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.854112 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.855495 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.855496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.855496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.855502 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.856484 (+ 982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.856736 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.856736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.856736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.856775 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.856885 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.856886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.856886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.856963 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.857001 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.857087 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.857144 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:17.857522 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:04:17.862437 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.857736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.0.1:52794)
0924 05:04:17.857798 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.857805 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.857843 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.857938 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.857939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.858049 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.858069 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.858070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.858603 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.858603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.859037 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.859038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.859337 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.859338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.859379 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.859645 (+ 266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.859648 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.860565 (+ 917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.861609 (+ 1044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.861610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.861610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.861782 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.861923 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.861923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.861924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.861971 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.862154 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.862155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.862187 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.862245 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.862323 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 05:04:17.862442 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.857770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.0.98.60:39645)
0924 05:04:17.857796 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.857798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.857832 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:04:17.857889 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.857890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.857898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.857937 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.858122 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.858545 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.859093 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.859287 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.859335 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.860635 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.860636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.860636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.860642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.861547 (+ 905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.861834 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.861834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.861835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.861877 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.862017 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.862018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.862018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.862102 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.862152 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.862268 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.862369 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":13}
I20260924 05:04:17.863564 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Bootstrap starting.
I20260924 05:04:17.864262 7213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:17.864494 7213 log.cc:824] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:17.865079 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: No bootstrap required, opened a new log
I20260924 05:04:17.866407 7213 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } }
I20260924 05:04:17.866523 7213 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:17.866542 7213 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd242b6afeff44eaa6c49cedc1eae3ea, State: Initialized, Role: FOLLOWER
I20260924 05:04:17.866633 7213 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } }
I20260924 05:04:17.866858 7218 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } } }
I20260924 05:04:17.866905 7213 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:17.866931 7218 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:04:17.868673 7229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:04:17.868727 7229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:04:17.870585 7145 master_runner.cc:428] Detected that this master bd242b6afeff44eaa6c49cedc1eae3ea is joining an existing cluster
I20260924 05:04:17.870621 7145 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39645
I20260924 05:04:17.877526 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.871770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.0.1:52796)
0924 05:04:17.871918 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.871926 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.871982 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.872116 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.872117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.872374 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.872432 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.872433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.873256 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.873257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.874115 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.874117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.874555 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.874613 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.874951 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.874955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.875221 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.876624 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.876625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.876625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.876779 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.876950 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.876951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.876951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.877005 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.877216 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.877217 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.877265 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.877314 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.877392 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260924 05:04:17.877583 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.871700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.0.98.60:39645)
0924 05:04:17.871730 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.871732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.871965 (+ 233us) server_negotiation.cc:407] Connection header received
0924 05:04:17.872039 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.872040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.872047 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.872072 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.872503 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.873185 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874195 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.874446 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874478 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.875292 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.875293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.875294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.875299 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.876555 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.876839 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.876839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.876840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.876890 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.877052 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.877053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.877053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.877164 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.877227 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.877346 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.877449 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":15}
I20260924 05:04:17.897625 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.871742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.0.98.61:45087)
0924 05:04:17.871776 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.871777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.872081 (+ 304us) server_negotiation.cc:407] Connection header received
0924 05:04:17.872161 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.872161 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.872171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.872197 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.872585 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.873280 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874266 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.874536 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874570 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.895090 (+ 20520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.895093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.895093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.895102 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.896501 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.896902 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.896903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.896904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.896963 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.897129 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.897131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.897131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.897269 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.897322 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.897447 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.897524 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:17.897639 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.871819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39854)
0924 05:04:17.872024 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.872041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.872100 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.872249 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.872250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.872468 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.872514 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.872515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.873356 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.873356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.874196 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.874197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.874630 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.874680 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.894346 (+ 19666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.894356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.894993 (+ 637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.896583 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.896584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.896585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.896774 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.897020 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.897021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.897021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.897080 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.897313 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.897313 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.897366 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.897424 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.897524 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":40,"thread_start_us":59,"threads_started":1}
I20260924 05:04:17.897812 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.871678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37222)
0924 05:04:17.871880 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.871889 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.871949 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.872100 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.872101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.872344 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.872407 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.872408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.873290 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.873291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.874236 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.874238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.874718 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.874776 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.894410 (+ 19634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.894414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.894993 (+ 579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.896836 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.896836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.896837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.897004 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.897170 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.897171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.897171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.897214 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.897476 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.897477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.897519 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.897577 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.897690 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":32,"thread_start_us":73,"threads_started":1}
I20260924 05:04:17.898101 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.871843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.98.62:42581)
0924 05:04:17.871878 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.871879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.871940 (+ 61us) server_negotiation.cc:407] Connection header received
0924 05:04:17.871995 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.871996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.872006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.872059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.872483 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.873204 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874325 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.874605 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.874672 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.895372 (+ 20700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.895373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.895374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.895382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.896748 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.897074 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.897075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.897076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.897122 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.897265 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.897266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.897267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.897392 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.897446 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.897940 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.898014 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:17.903188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6553
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:04:17.907584 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.900286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.0.98.60:39645)
0924 05:04:17.900324 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.900326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.900333 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:17.900362 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.900363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.900371 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.900425 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.900668 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.901122 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.901784 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.901972 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.902023 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.905075 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.905076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.905077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.905083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.906587 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.906918 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.906919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.906920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.906970 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.907137 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.907138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.907138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.907253 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.907311 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.907439 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.907503 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:17.907585 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.900222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.0.1:52804)
0924 05:04:17.900265 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.900274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.900308 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.900427 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.900427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.900572 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.900608 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.900610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.901186 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.901187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.901690 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.901691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.902024 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.902025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.902094 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.902411 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.902415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.904994 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.906663 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.906664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.906665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.906852 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.907025 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.907025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.907026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.907087 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.907307 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.907307 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.907351 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.907413 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.907499 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 05:04:17.908365 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39645: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39645
}
attrs {
promote: true
}
}
dest_uuid: "1a98c13313924495b97777015844226a"
cas_config_opid_index: 5
I20260924 05:04:17.908588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40019
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42581
--webserver_port=43935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:04:17.908603 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } }
I20260924 05:04:17.909013 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.909938 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39645
I20260924 05:04:17.910014 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.910080 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:17.910234 7145 master.cc:537] Master@127.0.98.60:39645 shutting down...
I20260924 05:04:17.910367 7187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:17.911218 7187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Refusing update from remote peer 1a98c13313924495b97777015844226a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 05:04:17.911921 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:04:17.912549 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a -> Peer bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60:39645): Couldn't send request to peer bd242b6afeff44eaa6c49cedc1eae3ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 05:04:17.916867 7145 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:17.916956 7145 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:17.917007 7145 tablet_replica.cc:330] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: stopping tablet replica
I20260924 05:04:17.917404 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.909887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39858)
0924 05:04:17.910716 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.910727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.910777 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.910971 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.910972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.910973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:17.911006 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.911007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.911826 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.911827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.913378 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.913381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.917173 (+ 3792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.917174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.917241 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.917262 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.917304 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.917339 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814}
I20260924 05:04:17.917639 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.909975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.98.61:45087)
0924 05:04:17.910686 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.910688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.910819 (+ 131us) server_negotiation.cc:407] Connection header received
0924 05:04:17.910841 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.910842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.910871 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.910918 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:17.911077 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.911741 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.913741 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.916994 (+ 3253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.917041 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.917117 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.917535 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.917573 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689}
I20260924 05:04:17.917845 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 LEARNER]: Refusing update from remote peer 1a98c13313924495b97777015844226a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 05:04:17.918010 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 05:04:17.918512 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.919775 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a98c13313924495b97777015844226a. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.919859 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: LEARNER
I20260924 05:04:17.919893 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: true } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.919939 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: LEARNER
I20260924 05:04:17.920151 7051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: attempting to promote NON_VOTER c18a88efbe12492d93b70836c35441e0 to VOTER
I20260924 05:04:17.920341 7051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } }
I20260924 05:04:17.920770 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 LEARNER]: Refusing update from remote peer 1a98c13313924495b97777015844226a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260924 05:04:17.920869 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a -> Peer bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60:39645): Couldn't send request to peer bd242b6afeff44eaa6c49cedc1eae3ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 05:04:17.920924 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 05:04:17.922134 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c18a88efbe12492d93b70836c35441e0 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.922935 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.923012 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:17.923054 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.923101 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:17.923365 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c18a88efbe12492d93b70836c35441e0 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.924289 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.924371 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:04:17.933192 7145 master.cc:552] Master@127.0.98.60:39645 shutdown complete.
I20260924 05:04:17.933252 7145 master_runner.cc:305] Clearing existing system tablet
I20260924 05:04:17.933369 7145 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.933413 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.934163 7145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:04:17.934659 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.934763 7145 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.934806 7145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.934826 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "8ca6740f507118451afaf0327f3911b9"
server_key_iv: "d187050c15bb2590592e07d665285bcf"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.935529 7145 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:17.936834 7145 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 05:04:17.936901 7145 log.cc:1196] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 05:04:17.937078 7145 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Deleting consensus metadata
I20260924 05:04:17.937209 7145 master_runner.cc:315] Copying system tablet from 127.0.98.62:42581
I20260924 05:04:17.937826 7145 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42581
I20260924 05:04:17.948405 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.938185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.0.98.62:42581)
0924 05:04:17.938232 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:17.938234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:17.941905 (+ 3671us) server_negotiation.cc:407] Connection header received
0924 05:04:17.941962 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:17.941963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:17.941977 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:17.942060 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:17.942394 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.943198 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.944198 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.944495 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.944566 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.945628 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:17.945629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:17.945630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:17.945637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:17.947256 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.947638 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.947640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.947640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.947700 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:17.947891 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:17.947891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:17.947892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:17.948015 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:17.948087 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:17.948239 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 05:04:17.948312 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:04:17.948426 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:17.938114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37238)
0924 05:04:17.941805 (+ 3691us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:17.941818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:17.941905 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:17.942066 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:17.942067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:17.942264 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:17.942315 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.942318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.943287 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.943288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.944110 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:17.944113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:17.944577 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:17.944578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:17.944646 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:17.945148 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:17.945153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:17.945539 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:17.947340 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.947341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.947342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.947555 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.947762 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:17.947763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:17.947763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:17.947831 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:17.948089 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:17.948090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:17.948138 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:04:17.948212 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:04:17.948312 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3664,"thread_start_us":64,"threads_started":1}
I20260924 05:04:17.948935 6511 tablet_copy_service.cc:140] P 1a98c13313924495b97777015844226a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bd242b6afeff44eaa6c49cedc1eae3ea ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37238)
I20260924 05:04:17.949003 6511 tablet_copy_service.cc:161] P 1a98c13313924495b97777015844226a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bd242b6afeff44eaa6c49cedc1eae3ea at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37238: session id = bd242b6afeff44eaa6c49cedc1eae3ea-00000000000000000000000000000000
I20260924 05:04:17.949592 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 05:04:17.950129 7145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 05:04:17.951455 7145 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: tablet copy: Starting download of 0 data blocks...
I20260924 05:04:17.951610 7145 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: tablet copy: Starting download of 1 WAL segments...
I20260924 05:04:17.952355 7145 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 05:04:17.953318 6511 tablet_copy_service.cc:342] P 1a98c13313924495b97777015844226a: Request end of tablet copy session bd242b6afeff44eaa6c49cedc1eae3ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37238
I20260924 05:04:17.953370 6511 tablet_copy_service.cc:433] P 1a98c13313924495b97777015844226a: ending tablet copy session bd242b6afeff44eaa6c49cedc1eae3ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bd242b6afeff44eaa6c49cedc1eae3ea
I20260924 05:04:17.955837 7145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:17.956856 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.956898 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:17.958419 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:17.958544 7145 server_base.cc:1034] running on GCE node
I20260924 05:04:17.958729 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:17.958866 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:17.959983 7145 hybrid_clock.cc:648] HybridClock initialized: now 1790226257959992 us; error 54 us; skew 500 ppm
Sep 24 05:04:17 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226257, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:17.961464 7145 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 05:04:17.961733 7145 webserver.cc:460] Webserver started at http://127.0.98.60:37099/ using document root <none> and password file <none>
I20260924 05:04:17.961843 7145 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:17.961874 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:17.962550 7145 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:17.963088 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:17.963330 7145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:17.963378 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "8ca6740f507118451afaf0327f3911b9"
server_key_iv: "d187050c15bb2590592e07d665285bcf"
server_key_version: "encryptionkey@0"
I20260924 05:04:17.963433 7145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:17.982290 7145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:17.984002 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:17.986974 7145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39645
I20260924 05:04:17.987068 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39645 every 8 connection(s)
I20260924 05:04:17.987295 7145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:04:17.988214 7312 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:04:17.988696 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Bootstrap starting.
I20260924 05:04:17.991588 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:04:17.991820 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Bootstrap complete.
I20260924 05:04:17.992013 7312 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } }
I20260924 05:04:17.992098 7312 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bd242b6afeff44eaa6c49cedc1eae3ea, State: Initialized, Role: LEARNER
I20260924 05:04:17.992132 7312 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } }
I20260924 05:04:17.992291 7312 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:17.993685 7326 catalog_manager.cc:1259] Loaded cluster ID: dde3d3ee0e5f402eaa284404d019683e
I20260924 05:04:17.993793 7326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: loading cluster ID for follower catalog manager: success
I20260924 05:04:17.993925 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:17.994014 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: This master's current role is: LEARNER
I20260924 05:04:17.995498 7326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: acquiring CA information for follower catalog manager: success
I20260924 05:04:17.995781 7326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 05:04:18.008944 7238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:18.009136 7238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:18.009164 7238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:18.010286 7238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:18.010324 7238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:18.010360 7238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:18.011799 7238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43935
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:18.012020 7238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:18.012212 7238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:18.016582 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:18.016597 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:18.016593 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:18.016772 7238 server_base.cc:1034] running on GCE node
I20260924 05:04:18.016938 7238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:18.017146 7238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:18.018276 7238 hybrid_clock.cc:648] HybridClock initialized: now 1790226258018263 us; error 26 us; skew 500 ppm
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:18.020476 7238 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:04:18.020849 7238 webserver.cc:460] Webserver started at http://127.0.98.1:43935/ using document root <none> and password file <none>
I20260924 05:04:18.021040 7238 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:18.021100 7238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:18.022341 7238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:04:18.022825 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.022964 7238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.023036 7238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "87e7066d8cc64d23b9e155c02ae63c82"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "18a9c37e396d62c41be2934323e5a210"
server_key_iv: "53dbea89ca93bafbb64e44f4768f453b"
server_key_version: "encryptionkey@0"
I20260924 05:04:18.023352 7238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:18.039001 7238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:18.040700 7238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:18.040833 7238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:18.041029 7238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:18.041309 7238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:18.041342 7238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.041369 7238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:18.041391 7238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.046856 7238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40019
I20260924 05:04:18.046914 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40019 every 8 connection(s)
I20260924 05:04:18.047211 7238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:04:18.054363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 7238
I20260924 05:04:18.054494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6687
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:04:18.057689 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.047981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55931 (local address 127.0.98.62:42581)
0924 05:04:18.048053 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.048055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.048234 (+ 179us) server_negotiation.cc:407] Connection header received
0924 05:04:18.048714 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.048715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.048727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.048755 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.049824 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.050597 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.051781 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.052037 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.052089 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.054300 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.054301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.054302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.054308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.056212 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.056676 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.056677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.056678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.056736 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.057004 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.057005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.057005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.057131 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.057172 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.057516 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.057584 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:04:18.058075 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.047821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:55931)
0924 05:04:18.048080 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.048122 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.048629 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.049298 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.049299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.049527 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.049746 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.049752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.050676 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.050677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.051707 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.051709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.052099 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.052100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.052171 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.052726 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.052731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.054222 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.056295 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.056301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.056303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.056569 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.056803 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.056804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.056804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.056928 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.057217 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.057223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.057373 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.057821 (+ 448us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.057909 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":64,"thread_start_us":78,"threads_started":1}
I20260924 05:04:18.058302 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.047887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35373 (local address 127.0.98.60:39645)
0924 05:04:18.048309 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.048311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.048319 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:18.048750 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.048751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.048763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.049341 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.049847 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.050524 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.051803 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.052045 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.052103 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.055254 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.055255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.055256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.055262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.056799 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.057157 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.057158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.057159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.057214 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.057366 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.057367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.057367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.057480 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.057531 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.058131 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.058214 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":51,"threads_started":1}
I20260924 05:04:18.058507 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42581
I20260924 05:04:18.058601 7456 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.058790 7456 heartbeater.cc:511] Master 127.0.98.62:42581 requested a full tablet report, sending...
I20260924 05:04:18.059132 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.047821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.98.1:45503)
0924 05:04:18.048080 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.048122 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.048606 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.048828 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.048832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.049546 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.049746 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.049752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.050689 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.050689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.051566 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.051568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.051961 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.051962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.052114 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.052786 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.052791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.055266 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.057480 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.057481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.057481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.057657 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.058528 (+ 871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.058529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.058529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.058587 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.058873 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.058874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.058909 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.058965 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.059046 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":96,"thread_start_us":76,"threads_started":1}
I20260924 05:04:18.059151 6491 ts_manager.cc:194] Re-registered known tserver with Master: 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019)
I20260924 05:04:18.059196 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.047885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45503 (local address 127.0.98.61:45087)
0924 05:04:18.047934 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.047936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.048213 (+ 277us) server_negotiation.cc:407] Connection header received
0924 05:04:18.048701 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.048702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.048714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.048762 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.049859 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.050601 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.051638 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.051890 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.051950 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.055360 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.055361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.055361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.055368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.057403 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.058051 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.058052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.058052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.058116 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.058661 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.058662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.058663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.058793 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.058843 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.059027 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.059108 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:04:18.059507 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55931
I20260924 05:04:18.058077 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.047821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.98.1:35373)
0924 05:04:18.048181 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.048189 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.048619 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.049364 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.049365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.049529 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.049746 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.049752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.050619 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.050620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.051725 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.051726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.052154 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.052154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.052206 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.052670 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.052678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.055167 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.056883 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.056884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.056884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.057086 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.057268 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.057268 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.057268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.057315 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.057553 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.057553 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.057592 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.057656 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.057841 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":217,"thread_start_us":61,"threads_started":1}
I20260924 05:04:18.060040 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45087
I20260924 05:04:18.060088 7457 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.060128 7457 heartbeater.cc:511] Master 127.0.98.61:45087 requested a full tablet report, sending...
I20260924 05:04:18.060425 7091 ts_manager.cc:194] Registered new tserver with Master: 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019)
I20260924 05:04:18.060899 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39645
I20260924 05:04:18.060942 7455 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.060992 7455 heartbeater.cc:511] Master 127.0.98.60:39645 requested a full tablet report, sending...
I20260924 05:04:18.061324 7268 ts_manager.cc:194] Registered new tserver with Master: 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019)
I20260924 05:04:18.061396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36027
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42581
--webserver_port=42937
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:18.139492 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:18.139695 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:18.139716 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:18.140785 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:18.140820 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:18.140843 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:18.142241 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42937
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:18.142416 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:18.142583 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:18.146904 7468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:18.146992 7469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:18.147094 7471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:18.147068 7463 server_base.cc:1034] running on GCE node
I20260924 05:04:18.147258 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:18.147437 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:18.148554 7463 hybrid_clock.cc:648] HybridClock initialized: now 1790226258148543 us; error 27 us; skew 500 ppm
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:18.150789 7463 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:04:18.151189 7463 webserver.cc:460] Webserver started at http://127.0.98.2:42937/ using document root <none> and password file <none>
I20260924 05:04:18.151367 7463 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:18.151423 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:18.152609 7463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:18.153071 7478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.153218 7463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.153292 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "344cc91000b8475bad832b85c56a1bb7"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "2287a27587f9babda96e00c31d9dc508"
server_key_iv: "2e0c23fb17e1963665a4a1d35ee43751"
server_key_version: "encryptionkey@0"
I20260924 05:04:18.153589 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:18.168169 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:18.169684 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:18.169822 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:18.170009 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:18.170264 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:18.170295 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.170320 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:18.170342 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.175589 7463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36027
I20260924 05:04:18.175649 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36027 every 8 connection(s)
I20260924 05:04:18.175925 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:18.185658 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.176527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56093 (local address 127.0.98.60:39645)
0924 05:04:18.176730 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.176732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.176786 (+ 54us) server_negotiation.cc:407] Connection header received
0924 05:04:18.177835 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.177837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.177848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.177879 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.178672 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.179407 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180399 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.180617 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180669 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.182784 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.182785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.182785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.182792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.184238 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.184667 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.184668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.184668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.184723 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.184923 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.184924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.184925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.185031 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.185084 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.185472 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.185555 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260924 05:04:18.185755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 7463
I20260924 05:04:18.185833 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.176537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44583 (local address 127.0.98.61:45087)
0924 05:04:18.176577 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.176578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.176769 (+ 191us) server_negotiation.cc:407] Connection header received
0924 05:04:18.177820 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.177821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.177832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.177875 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.178651 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.179400 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180221 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.180474 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180529 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.183139 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.183140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.183140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.183147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.184540 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.185090 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.185091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.185091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.185147 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.185326 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.185327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.185328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.185447 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.185499 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.185659 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.185755 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:18.185894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6821
I20260924 05:04:18.186172 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.176507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.98.2:56093)
0924 05:04:18.176655 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.176692 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.177698 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.177942 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.177947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.178396 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.178566 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.178572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.179478 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.179479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.180311 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.180312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.180676 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.180739 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.181246 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.181256 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.182704 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.184337 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.184343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.184345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.184600 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.184773 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.184774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.184774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.184871 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.185206 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.185210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.185325 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.185801 (+ 476us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.185962 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":33,"thread_start_us":79,"threads_started":1}
I20260924 05:04:18.186508 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.176941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48477 (local address 127.0.98.62:42581)
0924 05:04:18.176984 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.176985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.176994 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:18.177849 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.177851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.177861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.177889 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.178666 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.179152 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180365 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.180660 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180716 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.183983 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.183985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.183985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.183992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.185422 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.185788 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.185789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.185790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.185846 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.186028 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.186029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.186030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.186142 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.186185 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.186356 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.186425 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:04:18.186745 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.176507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.98.2:44583)
0924 05:04:18.176673 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.176693 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.177704 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.177941 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.177945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.178380 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.178565 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.178570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.179503 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.179504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.180147 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.180149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.180586 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.180714 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.181273 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.181277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.183057 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.184616 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.184617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.184618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.185022 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.185203 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.185204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.185204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.185263 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.185523 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.185524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.185559 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.186456 (+ 897us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.186551 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":5,"thread_start_us":98,"threads_started":1}
I20260924 05:04:18.186816 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.176791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:48477)
0924 05:04:18.176919 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.176928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.177732 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.177950 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.177951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.178404 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.178566 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.178572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.179242 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.179242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.180280 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.180282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.180763 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.180764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.180814 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.181418 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.181422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.183591 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.185475 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.185476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.185476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.185677 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.185913 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.185913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.185914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.185968 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.186212 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.186213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.186260 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.186592 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.186685 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":21,"thread_start_us":62,"threads_started":1}
I20260924 05:04:18.187245 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42581
I20260924 05:04:18.187326 7593 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.187484 7593 heartbeater.cc:511] Master 127.0.98.62:42581 requested a full tablet report, sending...
I20260924 05:04:18.187800 6491 ts_manager.cc:194] Re-registered known tserver with Master: 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:18.188153 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48477
I20260924 05:04:18.189957 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39645
I20260924 05:04:18.190006 7592 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.190065 7592 heartbeater.cc:511] Master 127.0.98.60:39645 requested a full tablet report, sending...
I20260924 05:04:18.190161 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45087
I20260924 05:04:18.190209 7594 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.190250 7594 heartbeater.cc:511] Master 127.0.98.61:45087 requested a full tablet report, sending...
I20260924 05:04:18.190317 7268 ts_manager.cc:194] Registered new tserver with Master: 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:18.190515 7091 ts_manager.cc:194] Registered new tserver with Master: 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:18.191319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36241
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42581
--webserver_port=46285
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:18.269789 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:18.269977 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:18.270005 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:18.271088 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:18.271126 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:18.271159 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:18.272554 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46285
--tserver_master_addrs=127.0.98.62:42581,127.0.98.61:45087,127.0.98.60:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:18.272783 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:18.272980 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:18.277448 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:18.277455 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:18.277454 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:18.277603 7599 server_base.cc:1034] running on GCE node
I20260924 05:04:18.277829 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:18.278045 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:18.279176 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790226258279146 us; error 42 us; skew 500 ppm
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:18.281358 7599 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:04:18.281769 7599 webserver.cc:460] Webserver started at http://127.0.98.3:46285/ using document root <none> and password file <none>
I20260924 05:04:18.281960 7599 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:18.282017 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:18.283190 7599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:04:18.283622 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.283763 7599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.283836 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "9cf97fa9703e45819ac3330ab207cac5"
format_stamp: "Formatted at 2026-09-24 05:04:17 on dist-test-slave-m01t"
server_key: "a8dc1bacc9739b58ed548089f1900f45"
server_key_iv: "b707e1db7d080dd103a5d3b0526b884c"
server_key_version: "encryptionkey@0"
I20260924 05:04:18.284124 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:18.298224 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:18.300007 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:18.300135 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:18.300338 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:18.300630 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:18.300661 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.300688 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:18.300711 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:18.305770 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36241
I20260924 05:04:18.305833 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36241 every 8 connection(s)
I20260924 05:04:18.306113 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 05:04:18 dist-test-slave-m01t krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226258, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:04:18.315582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 7599
I20260924 05:04:18.316157 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.306970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51101 (local address 127.0.98.61:45087)
0924 05:04:18.307016 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.307018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.307093 (+ 75us) server_negotiation.cc:407] Connection header received
0924 05:04:18.308241 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.308242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.308250 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.308291 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.308893 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.309644 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311380 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.311554 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311604 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.313430 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.313431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.313431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.313437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.314762 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315170 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.315172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.315172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.315229 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315489 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.315490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.315490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.315618 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.315667 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.316004 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.316072 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:18.316586 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.306774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43285 (local address 127.0.98.62:42581)
0924 05:04:18.307024 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.307026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.307148 (+ 122us) server_negotiation.cc:407] Connection header received
0924 05:04:18.308135 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.308137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.308148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.308196 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.309753 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.310203 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.310943 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.311220 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311282 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.313926 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.313928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.313928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.313935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.315528 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315871 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.315872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.315872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.315922 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.316101 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.316102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.316102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.316222 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.316263 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.316444 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.316508 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260924 05:04:18.316601 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.306722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.98.3:51101)
0924 05:04:18.306973 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.307011 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.307965 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.308308 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.308309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.308600 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.308810 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.308814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.309763 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.309764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.310923 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.310924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.311617 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.311682 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.312024 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.312027 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.313353 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.314823 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.314852 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.314854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.315105 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.315287 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.315288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.315405 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.315710 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.315715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.315860 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.316236 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.316391 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":75,"spinlock_wait_cycles":6912,"thread_start_us":90,"threads_started":1}
I20260924 05:04:18.316978 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.306824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50483 (local address 127.0.98.60:39645)
0924 05:04:18.306865 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.306867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.307250 (+ 383us) server_negotiation.cc:407] Connection header received
0924 05:04:18.308057 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.308058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.308069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.308135 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.308922 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.309651 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.310989 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.311280 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311336 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.314396 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.314397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.314398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.314404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.315854 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.316216 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.316217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.316217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.316273 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.316471 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.316472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.316472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.316595 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.316638 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.316824 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.316895 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:18.316998 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45087
I20260924 05:04:18.317090 7730 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.317243 7730 heartbeater.cc:511] Master 127.0.98.61:45087 requested a full tablet report, sending...
I20260924 05:04:18.317667 7091 ts_manager.cc:194] Registered new tserver with Master: 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241)
I20260924 05:04:18.317996 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.306722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.98.3:50483)
0924 05:04:18.307208 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.307215 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.307977 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.308213 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.308217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.308604 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.308809 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.308815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.309763 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.309764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.310919 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.310921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.311339 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.311408 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.312239 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.312243 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.314299 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.315937 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.315938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.316132 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.316346 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.316346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.316347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.316406 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.316684 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.316685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.316726 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.317093 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.317185 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":54,"thread_start_us":84,"threads_started":1}
I20260924 05:04:18.318277 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39645
I20260924 05:04:18.318327 7728 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.318385 7728 heartbeater.cc:511] Master 127.0.98.60:39645 requested a full tablet report, sending...
I20260924 05:04:18.318635 7268 ts_manager.cc:194] Registered new tserver with Master: 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241)
I20260924 05:04:18.316975 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.306722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:43285)
0924 05:04:18.307100 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.307108 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.307975 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.308213 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.308217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.308604 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:18.308810 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.308815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.310272 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.310273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.310871 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.310872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.311287 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.311288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.311408 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.311916 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:18.311926 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:18.313832 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:18.315595 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.315596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.315796 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.315987 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:18.315987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:18.315988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:18.316037 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:18.316304 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:18.316305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:18.316339 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.316797 (+ 458us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.316885 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260924 05:04:18.319805 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42581
I20260924 05:04:18.319859 7729 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:18.319922 7729 heartbeater.cc:511] Master 127.0.98.62:42581 requested a full tablet report, sending...
I20260924 05:04:18.320134 6491 ts_manager.cc:194] Re-registered known tserver with Master: 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241)
I20260924 05:04:18.320603 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43285
I20260924 05:04:18.324990 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.317653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.98.62:42581)
0924 05:04:18.317682 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.317683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.318051 (+ 368us) server_negotiation.cc:407] Connection header received
0924 05:04:18.318141 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.318142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.318153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.318179 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.318822 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.319478 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.321148 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.321413 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.321445 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.322657 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.322658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.322659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.322665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.324074 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.324353 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.324354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.324354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.324404 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.324556 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.324557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.324557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.324656 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.324733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.324828 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.324895 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:04:18.325040 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.317437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.0.98.61:45087)
0924 05:04:18.317469 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.317470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.317633 (+ 163us) server_negotiation.cc:407] Connection header received
0924 05:04:18.317750 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.317751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.317761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.317791 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.318547 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.319256 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.320208 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.320501 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.320540 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.322055 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.322056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.322057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.322062 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.323395 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.323740 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.323741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.323741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.323794 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.324490 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.324491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.324491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.324627 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.324697 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.324885 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.324955 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:04:18.325395 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.317397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.98.60:39645)
0924 05:04:18.317429 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.317431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.317924 (+ 493us) server_negotiation.cc:407] Connection header received
0924 05:04:18.318007 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.318008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.318017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.318043 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.319203 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.319647 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.321256 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.321533 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.321573 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.322971 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.322972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.322973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.322979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.324271 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.324623 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.324624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.324625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.324672 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.324858 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.324859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.324859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.324970 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.325048 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.325244 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.325312 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:04:18.326676 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:18.328233 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:18.335498 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.331615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36241 (local address 127.0.0.1:53422)
0924 05:04:18.331815 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.331826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.331872 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.332383 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.332384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.332386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:18.332434 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.332435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.333353 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.333353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.334260 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.334263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335259 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.335260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.335323 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.335328 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.335371 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.335413 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":2}
I20260924 05:04:18.335801 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.331702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.0.98.3:36241)
0924 05:04:18.331974 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.331978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.331986 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:18.332016 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.332018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.332030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.332139 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:18.332505 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.333292 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.334346 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335092 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.335148 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.335229 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.335677 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.335720 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":75,"threads_started":1}
I20260924 05:04:18.336062 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.331694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40019 (local address 127.0.0.1:56204)
0924 05:04:18.332014 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.332022 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.332062 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.332358 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.332359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.332363 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:18.332432 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.332435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.333747 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.333748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.334866 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.334868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335836 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.335838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.335897 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.335901 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.335949 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.335989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":88,"threads_started":1}
I20260924 05:04:18.336170 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.331646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36027 (local address 127.0.0.1:50858)
0924 05:04:18.331820 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.331830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.331867 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.332910 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.332910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.332912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:18.332942 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.332943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.334104 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.334105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.335127 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.335129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335999 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.335999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.336053 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.336056 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.336082 (+ 26us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.336115 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 05:04:18.336515 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.332001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.0.98.1:40019)
0924 05:04:18.332186 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.332191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.332200 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:18.332238 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.332240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.332255 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.332336 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:18.332549 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.333638 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.334953 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335664 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.335733 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.335803 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.336232 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.336277 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260924 05:04:18.336472 7662 tablet_service.cc:1467] Processing CreateTablet for tablet cd6ffb986c0f41f98b98134adfd77315 (DEFAULT_TABLE table=test-table [id=6cf44d9a0c5a4bf0abd39cf138b16316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:18.336774 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.331743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.98.2:36027)
0924 05:04:18.332754 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.332759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.332769 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:18.332804 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.332806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.332821 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.332898 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:18.333000 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.334019 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.335216 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335921 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.336299 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.336635 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.336663 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.336702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":74,"threads_started":1}
I20260924 05:04:18.336877 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6ffb986c0f41f98b98134adfd77315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:18.337076 7389 tablet_service.cc:1467] Processing CreateTablet for tablet cd6ffb986c0f41f98b98134adfd77315 (DEFAULT_TABLE table=test-table [id=6cf44d9a0c5a4bf0abd39cf138b16316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:18.337317 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6ffb986c0f41f98b98134adfd77315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:18.338337 7526 tablet_service.cc:1467] Processing CreateTablet for tablet cd6ffb986c0f41f98b98134adfd77315 (DEFAULT_TABLE table=test-table [id=6cf44d9a0c5a4bf0abd39cf138b16316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:18.338650 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6ffb986c0f41f98b98134adfd77315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:18.339845 7752 tablet_bootstrap.cc:492] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Bootstrap starting.
I20260924 05:04:18.340265 7753 tablet_bootstrap.cc:492] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Bootstrap starting.
I20260924 05:04:18.340430 7752 tablet_bootstrap.cc:654] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:18.340727 7752 log.cc:824] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:18.340837 7753 tablet_bootstrap.cc:654] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:18.341074 7753 log.cc:824] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:18.341439 7752 tablet_bootstrap.cc:492] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: No bootstrap required, opened a new log
I20260924 05:04:18.341491 7752 ts_tablet_manager.cc:1397] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:18.341641 7756 tablet_bootstrap.cc:492] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Bootstrap starting.
I20260924 05:04:18.341851 7753 tablet_bootstrap.cc:492] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: No bootstrap required, opened a new log
I20260924 05:04:18.341939 7753 ts_tablet_manager.cc:1397] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:18.342346 7756 tablet_bootstrap.cc:654] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:18.342617 7756 log.cc:824] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:18.343222 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.334324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.0.1:52822)
0924 05:04:18.334754 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.334764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.334828 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.334997 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.334997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.334999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:18.335046 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.335047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.336461 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.336462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.342122 (+ 5660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.342125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.342997 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.342997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.343065 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.343068 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.343109 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.343149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":78,"threads_started":1}
I20260924 05:04:18.343219 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.334494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.0.98.60:39645)
0924 05:04:18.334634 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.334636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.334806 (+ 170us) server_negotiation.cc:407] Connection header received
0924 05:04:18.334889 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.334890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.334910 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.334944 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:18.335117 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.335751 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.342222 (+ 6471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.342803 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.342850 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.342903 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.343124 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.343159 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 05:04:18.343758 7286 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 05:04:18.343746 7753 raft_consensus.cc:348] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.343895 7753 raft_consensus.cc:374] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:18.343936 7753 raft_consensus.cc:729] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf97fa9703e45819ac3330ab207cac5, State: Initialized, Role: FOLLOWER
I20260924 05:04:18.343864 7752 raft_consensus.cc:348] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.343982 7752 raft_consensus.cc:374] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:18.344015 7752 raft_consensus.cc:729] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e7066d8cc64d23b9e155c02ae63c82, State: Initialized, Role: FOLLOWER
I20260924 05:04:18.344061 7753 consensus_queue.cc:260] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.344105 7752 consensus_queue.cc:260] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.344314 7752 ts_tablet_manager.cc:1428] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:04:18.344314 7753 ts_tablet_manager.cc:1428] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:18.344398 7729 heartbeater.cc:503] Master 127.0.98.62:42581 was elected leader, sending a full tablet report...
I20260924 05:04:18.344559 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a98c13313924495b97777015844226a. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:18.344619 7456 heartbeater.cc:503] Master 127.0.98.62:42581 was elected leader, sending a full tablet report...
I20260924 05:04:18.344648 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: This master's current role is: LEARNER
I20260924 05:04:18.345763 7756 tablet_bootstrap.cc:492] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: No bootstrap required, opened a new log
I20260924 05:04:18.345825 7756 ts_tablet_manager.cc:1397] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 05:04:18.347299 7756 raft_consensus.cc:348] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.347414 7756 raft_consensus.cc:374] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:18.347434 7756 raft_consensus.cc:729] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344cc91000b8475bad832b85c56a1bb7, State: Initialized, Role: FOLLOWER
I20260924 05:04:18.347517 7756 consensus_queue.cc:260] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.347714 7756 ts_tablet_manager.cc:1428] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:18.347765 7593 heartbeater.cc:503] Master 127.0.98.62:42581 was elected leader, sending a full tablet report...
I20260924 05:04:18.385141 7759 raft_consensus.cc:482] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:18.385236 7759 raft_consensus.cc:504] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.385622 7759 leader_election.cc:288] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241), 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:18.388276 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.385815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33149 (local address 127.0.98.3:36241)
0924 05:04:18.385855 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.385857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.385915 (+ 58us) server_negotiation.cc:407] Connection header received
0924 05:04:18.385964 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.385965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.385976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.386020 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:18.386169 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.386646 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.387627 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.387978 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.388035 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.388046 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.388182 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.388213 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:18.388278 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.385776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36241 (local address 127.0.98.1:33149)
0924 05:04:18.385858 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.385867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.385916 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.386025 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.386025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.386027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:18.386116 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.386117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.386710 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.386710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.387548 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.387550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.388049 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.388050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.388124 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.388127 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.388171 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.388209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":38}
I20260924 05:04:18.388324 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.385832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56905 (local address 127.0.98.2:36027)
0924 05:04:18.385871 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.385873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.385904 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:04:18.385962 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.385962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.385972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.386015 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:18.386135 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.386649 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.387686 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.388038 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.388093 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.388102 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.388231 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.388265 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:18.388368 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.385807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36027 (local address 127.0.98.1:56905)
0924 05:04:18.385868 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:18.385877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:18.385917 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:18.386017 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:18.386017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:18.386019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:18.386082 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.386084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.386715 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.386715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.387602 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:18.387604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.388099 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.388100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:18.388177 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.388180 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:18.388222 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:18.388261 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":30}
I20260924 05:04:18.388859 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6ffb986c0f41f98b98134adfd77315" candidate_uuid: "87e7066d8cc64d23b9e155c02ae63c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344cc91000b8475bad832b85c56a1bb7" is_pre_election: true
I20260924 05:04:18.389022 7546 raft_consensus.cc:2454] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e7066d8cc64d23b9e155c02ae63c82 in term 0.
I20260924 05:04:18.389237 7342 leader_election.cc:302] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344cc91000b8475bad832b85c56a1bb7, 87e7066d8cc64d23b9e155c02ae63c82; no voters:
I20260924 05:04:18.389292 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6ffb986c0f41f98b98134adfd77315" candidate_uuid: "87e7066d8cc64d23b9e155c02ae63c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf97fa9703e45819ac3330ab207cac5" is_pre_election: true
I20260924 05:04:18.389360 7759 raft_consensus.cc:2781] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:18.389427 7759 raft_consensus.cc:482] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:18.389433 7682 raft_consensus.cc:2454] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e7066d8cc64d23b9e155c02ae63c82 in term 0.
I20260924 05:04:18.389456 7759 raft_consensus.cc:3037] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:18.390168 7759 raft_consensus.cc:504] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.390340 7759 leader_election.cc:288] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241), 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:18.390395 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6ffb986c0f41f98b98134adfd77315" candidate_uuid: "87e7066d8cc64d23b9e155c02ae63c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344cc91000b8475bad832b85c56a1bb7"
I20260924 05:04:18.390401 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6ffb986c0f41f98b98134adfd77315" candidate_uuid: "87e7066d8cc64d23b9e155c02ae63c82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf97fa9703e45819ac3330ab207cac5"
I20260924 05:04:18.390466 7546 raft_consensus.cc:3037] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:18.390470 7682 raft_consensus.cc:3037] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:18.391110 7682 raft_consensus.cc:2454] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e7066d8cc64d23b9e155c02ae63c82 in term 1.
I20260924 05:04:18.391115 7546 raft_consensus.cc:2454] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e7066d8cc64d23b9e155c02ae63c82 in term 1.
I20260924 05:04:18.391260 7342 leader_election.cc:302] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344cc91000b8475bad832b85c56a1bb7, 87e7066d8cc64d23b9e155c02ae63c82; no voters:
I20260924 05:04:18.391330 7759 raft_consensus.cc:2781] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:18.391459 7759 raft_consensus.cc:686] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 LEADER]: Becoming Leader. State: Replica: 87e7066d8cc64d23b9e155c02ae63c82, State: Running, Role: LEADER
I20260924 05:04:18.391579 7759 consensus_queue.cc:237] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:18.392496 6490 catalog_manager.cc:5495] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "87e7066d8cc64d23b9e155c02ae63c82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:18.392987 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: true } } }
I20260924 05:04:18.393083 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:18.393841 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: attempting to promote NON_VOTER bd242b6afeff44eaa6c49cedc1eae3ea to VOTER
I20260924 05:04:18.393957 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:18.394337 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Refusing update from remote peer 1a98c13313924495b97777015844226a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 05:04:18.394495 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 LEARNER]: Refusing update from remote peer 1a98c13313924495b97777015844226a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 05:04:18.394546 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 05:04:18.394932 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 05:04:18.395056 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.395181 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.395605 7286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.395939 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.395943 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.395999 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:18.396016 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:04:18.396052 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.396085 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:18.396132 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.396173 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:18.396562 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a98c13313924495b97777015844226a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:18.396665 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:04:18.411965 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.409828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.98.1:40019)
0924 05:04:18.409866 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.409868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.409876 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:18.409946 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.409946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.409966 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.410023 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:18.410137 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.410606 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.411325 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.411519 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.411569 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.411707 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:18.411799 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:18.411866 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.411884 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.411903 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:18.413448 7546 raft_consensus.cc:1261] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Refusing update from remote peer 87e7066d8cc64d23b9e155c02ae63c82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:18.413463 7682 raft_consensus.cc:1261] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Refusing update from remote peer 87e7066d8cc64d23b9e155c02ae63c82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:18.413754 7765 consensus_queue.cc:1035] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:18.413866 7759 consensus_queue.cc:1035] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:18.417097 7768 mvcc.cc:205] Tried to move back new op lower bound from 7332766754459308032 to 7332766754372468736. Current Snapshot: MvccSnapshot[applied={T|T < 7332766754459308032}]
I20260924 05:04:18.417194 7770 mvcc.cc:205] Tried to move back new op lower bound from 7332766754459308032 to 7332766754372468736. Current Snapshot: MvccSnapshot[applied={T|T < 7332766754459308032 or (T in {7332766754459308032})}]
I20260924 05:04:18.417778 7769 mvcc.cc:205] Tried to move back new op lower bound from 7332766754459308032 to 7332766754372468736. Current Snapshot: MvccSnapshot[applied={T|T < 7332766754459308032}]
I20260924 05:04:18.419298 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:18.419377 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:18.420531 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a98c13313924495b97777015844226a: Sending DeleteTablet for 3 replicas of tablet cd6ffb986c0f41f98b98134adfd77315
I20260924 05:04:18.420882 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6ffb986c0f41f98b98134adfd77315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53422
I20260924 05:04:18.420897 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6ffb986c0f41f98b98134adfd77315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56204
I20260924 05:04:18.421234 7526 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6ffb986c0f41f98b98134adfd77315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50858
I20260924 05:04:18.421424 7783 tablet_replica.cc:330] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: stopping tablet replica
I20260924 05:04:18.421514 7783 raft_consensus.cc:2229] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:18.421609 7783 raft_consensus.cc:2258] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:18.421692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6459
I20260924 05:04:18.422025 7783 ts_tablet_manager.cc:1905] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:18.423550 7783 ts_tablet_manager.cc:1918] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:18.423619 7783 log.cc:1196] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/cd6ffb986c0f41f98b98134adfd77315
I20260924 05:04:18.423866 7783 ts_tablet_manager.cc:1939] T cd6ffb986c0f41f98b98134adfd77315 P 344cc91000b8475bad832b85c56a1bb7: Deleting consensus metadata
I20260924 05:04:18.425981 7780 tablet_replica.cc:330] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: stopping tablet replica
I20260924 05:04:18.426144 7780 raft_consensus.cc:2229] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:18.426280 7780 raft_consensus.cc:2258] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:18.426452 7781 tablet_replica.cc:330] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: stopping tablet replica
I20260924 05:04:18.426599 7781 raft_consensus.cc:2229] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:18.426721 7781 raft_consensus.cc:2258] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:18.427454 7781 ts_tablet_manager.cc:1905] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:18.428545 7780 ts_tablet_manager.cc:1905] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:04:18.428640 7482 connection.cc:535] server connection from 127.0.0.1:50858 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:04:18.429436 7781 ts_tablet_manager.cc:1918] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:18.429563 7781 log.cc:1196] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/cd6ffb986c0f41f98b98134adfd77315
I20260924 05:04:18.429889 7781 ts_tablet_manager.cc:1939] T cd6ffb986c0f41f98b98134adfd77315 P 87e7066d8cc64d23b9e155c02ae63c82: Deleting consensus metadata
I20260924 05:04:18.430096 7780 ts_tablet_manager.cc:1918] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:18.430179 7780 log.cc:1196] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/cd6ffb986c0f41f98b98134adfd77315
W20260924 05:04:18.430378 7342 connection.cc:441] server connection from 127.0.0.1:56204 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56204 (request call id 1) could send its response
I20260924 05:04:18.430431 7780 ts_tablet_manager.cc:1939] T cd6ffb986c0f41f98b98134adfd77315 P 9cf97fa9703e45819ac3330ab207cac5: Deleting consensus metadata
W20260924 05:04:18.430758 7615 connection.cc:441] server connection from 127.0.0.1:53422 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53422 (request call id 1) could send its response
I20260924 05:04:18.431887 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.421381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.98.60:39645)
0924 05:04:18.421419 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.421421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.421579 (+ 158us) server_negotiation.cc:407] Connection header received
0924 05:04:18.421668 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.421668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.421678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.421705 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.422956 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.423679 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.424642 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.424913 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.424950 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.426658 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.426660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.426661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.426668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.428376 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.431068 (+ 2692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.431070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.431070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.431127 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.431317 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.431318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.431318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.431445 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.431523 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.431720 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.431797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:18.437875 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.430563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.98.61:45087)
0924 05:04:18.430606 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.430607 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.430802 (+ 195us) server_negotiation.cc:407] Connection header received
0924 05:04:18.430968 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.430969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.430980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.431008 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.431594 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.432317 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.433403 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.433696 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.433755 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.435119 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.435120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.435121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.435129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.436657 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.437024 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.437025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.437025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.437092 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.437248 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.437249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.437249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.437377 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.437464 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.437665 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.437770 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:18.438529 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:18.431002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.0.98.60:39645)
0924 05:04:18.431417 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:18.431419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:18.431575 (+ 156us) server_negotiation.cc:407] Connection header received
0924 05:04:18.431594 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:18.431595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:18.431605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:18.431659 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:18.432300 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.433057 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.433952 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:18.434255 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:18.434309 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:18.435505 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:18.435506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:18.435507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:18.435513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:18.436964 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.437302 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.437303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.437303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.437352 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:18.437527 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:18.437527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:18.437528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:18.437652 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:18.437755 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:18.438361 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 05:04:18.438440 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":72,"threads_started":1}
I20260924 05:04:19.417428 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.417216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:56353)
0924 05:04:19.417316 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.417380 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
W20260924 05:04:19.417811 7729 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42581 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.418216 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.418037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:54273)
0924 05:04:19.418128 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.418184 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":44,"threads_started":1}
W20260924 05:04:19.418386 7593 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42581 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.418918 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.418741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:49661)
0924 05:04:19.418834 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.418885 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
W20260924 05:04:19.419083 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42581 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.920840 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a98c13313924495b97777015844226a)
I20260924 05:04:19.920985 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a98c13313924495b97777015844226a)
I20260924 05:04:19.920982 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:19.921108 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:19.921239 7795 leader_election.cc:288] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a98c13313924495b97777015844226a (127.0.98.62:42581), c18a88efbe12492d93b70836c35441e0 (127.0.98.61:45087)
I20260924 05:04:19.921339 7796 leader_election.cc:288] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a98c13313924495b97777015844226a (127.0.98.62:42581), bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60:39645)
I20260924 05:04:19.921578 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.921363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37714)
0924 05:04:19.921486 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.921534 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260924 05:04:19.921595 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.921419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37720)
0924 05:04:19.921515 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.921559 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
W20260924 05:04:19.921687 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
W20260924 05:04:19.921788 7260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.922163 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.922009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37732)
0924 05:04:19.922068 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.922107 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 05:04:19.922292 7075 leader_election.cc:334] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.922425 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.922262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37734)
0924 05:04:19.922374 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.922401 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 05:04:19.922489 7260 leader_election.cc:334] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.924319 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.921373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45087 (local address 127.0.0.1:39890)
0924 05:04:19.921734 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.921744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:19.921802 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:19.921999 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:19.922001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:19.922003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:19.922063 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:19.922065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.922868 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.922869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:19.923612 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:19.923614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.924103 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.924104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:19.924165 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:19.924167 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:19.924211 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:04:19.924253 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":56,"threads_started":1}
I20260924 05:04:19.924314 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.921511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.98.61:45087)
0924 05:04:19.921856 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:19.921857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:19.921864 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:19.921891 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:19.921891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:19.921901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:19.922123 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:19.922154 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.922794 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.923689 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.924041 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.924098 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:19.924109 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:19.924220 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 05:04:19.924252 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":40,"threads_started":1}
I20260924 05:04:19.924464 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.921510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39645 (local address 127.0.0.1:52838)
0924 05:04:19.921893 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.921902 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:19.921961 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:19.922116 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:19.922117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:19.922119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:19.922168 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:19.922170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.922914 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.922915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:19.923672 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:19.923674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.924201 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.924202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:19.924261 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:19.924264 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:19.924307 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:04:19.924348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":37,"threads_started":1}
I20260924 05:04:19.924526 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c18a88efbe12492d93b70836c35441e0" is_pre_election: true
I20260924 05:04:19.924592 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.921692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.0.98.60:39645)
0924 05:04:19.921811 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:19.921813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:19.922009 (+ 196us) server_negotiation.cc:407] Connection header received
0924 05:04:19.922033 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:19.922034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:19.922042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:19.922095 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:19.922246 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.922821 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.923752 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:19.924131 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:19.924190 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:19.924201 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:19.924315 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 05:04:19.924477 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260924 05:04:19.924638 7101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd242b6afeff44eaa6c49cedc1eae3ea in term 1.
I20260924 05:04:19.924762 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c18a88efbe12492d93b70836c35441e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" is_pre_election: true
I20260924 05:04:19.924816 7263 leader_election.cc:302] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd242b6afeff44eaa6c49cedc1eae3ea, c18a88efbe12492d93b70836c35441e0; no voters: 1a98c13313924495b97777015844226a
I20260924 05:04:19.924855 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18a88efbe12492d93b70836c35441e0 in term 1.
I20260924 05:04:19.924942 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 05:04:19.924975 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a98c13313924495b97777015844226a)
I20260924 05:04:19.924968 7076 leader_election.cc:302] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd242b6afeff44eaa6c49cedc1eae3ea, c18a88efbe12492d93b70836c35441e0; no voters: 1a98c13313924495b97777015844226a
I20260924 05:04:19.924996 7795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:04:19.925052 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 05:04:19.925091 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a98c13313924495b97777015844226a)
I20260924 05:04:19.925107 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:04:19.925671 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:19.925735 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:19.925807 7795 leader_election.cc:288] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 2 election: Requested vote from peers 1a98c13313924495b97777015844226a (127.0.98.62:42581), c18a88efbe12492d93b70836c35441e0 (127.0.98.61:45087)
I20260924 05:04:19.925998 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.925920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37746)
0924 05:04:19.925954 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.925975 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:04:19.925984 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c18a88efbe12492d93b70836c35441e0"
I20260924 05:04:19.926061 7101 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bd242b6afeff44eaa6c49cedc1eae3ea in current term 2: Already voted for candidate c18a88efbe12492d93b70836c35441e0 in this term.
I20260924 05:04:19.926126 7796 leader_election.cc:288] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [CANDIDATE]: Term 2 election: Requested vote from peers 1a98c13313924495b97777015844226a (127.0.98.62:42581), bd242b6afeff44eaa6c49cedc1eae3ea (127.0.98.60:39645)
I20260924 05:04:19.926115 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c18a88efbe12492d93b70836c35441e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea"
I20260924 05:04:19.926185 7286 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c18a88efbe12492d93b70836c35441e0 in current term 2: Already voted for candidate bd242b6afeff44eaa6c49cedc1eae3ea in this term.
I20260924 05:04:19.926294 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.925914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37758)
0924 05:04:19.925948 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.925977 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 05:04:19.926323 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.926246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37768)
0924 05:04:19.926277 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.926303 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 05:04:19.926426 7260 leader_election.cc:334] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.926470 7260 leader_election.cc:302] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd242b6afeff44eaa6c49cedc1eae3ea; no voters: 1a98c13313924495b97777015844226a, c18a88efbe12492d93b70836c35441e0
I20260924 05:04:19.926596 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 05:04:19.926632 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:19.926559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:37770)
0924 05:04:19.926587 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:19.926611 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 05:04:19.926676 7075 leader_election.cc:334] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:19.926712 7075 leader_election.cc:302] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18a88efbe12492d93b70836c35441e0; no voters: 1a98c13313924495b97777015844226a, bd242b6afeff44eaa6c49cedc1eae3ea
I20260924 05:04:19.926779 7796 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 05:04:20.418462 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:20.418168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:38949)
0924 05:04:20.418360 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:20.418406 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260924 05:04:20.418857 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:20.418652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:54215)
0924 05:04:20.418763 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:20.418795 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260924 05:04:20.419456 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:20.419312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:47947)
0924 05:04:20.419401 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:20.419423 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260924 05:04:21.419225 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.418901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:47425)
0924 05:04:21.419128 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.419171 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":60,"threads_started":1}
I20260924 05:04:21.419441 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.419247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:33055)
0924 05:04:21.419363 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.419395 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260924 05:04:21.419951 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.419800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:48421)
0924 05:04:21.419888 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.419917 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":46,"threads_started":1}
I20260924 05:04:21.467052 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:21.467140 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:21.467278 7811 leader_election.cc:288] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1a98c13313924495b97777015844226a (127.0.98.62:42581), c18a88efbe12492d93b70836c35441e0 (127.0.98.61:45087)
I20260924 05:04:21.467470 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c18a88efbe12492d93b70836c35441e0" is_pre_election: true
I20260924 05:04:21.467582 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.467400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:38016)
0924 05:04:21.467499 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.467541 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260924 05:04:21.467579 7101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd242b6afeff44eaa6c49cedc1eae3ea in term 2.
I20260924 05:04:21.467724 7263 leader_election.cc:302] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd242b6afeff44eaa6c49cedc1eae3ea, c18a88efbe12492d93b70836c35441e0; no voters:
I20260924 05:04:21.467804 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 05:04:21.467836 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:21.467861 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:04:21.467962 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.467870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:38024)
0924 05:04:21.467913 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.467942 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:04:21.468032 7260 leader_election.cc:334] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:21.468573 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:21.468689 7811 leader_election.cc:288] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 3 election: Requested vote from peers 1a98c13313924495b97777015844226a (127.0.98.62:42581), c18a88efbe12492d93b70836c35441e0 (127.0.98.61:45087)
I20260924 05:04:21.468852 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.468770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:38032)
0924 05:04:21.468800 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.468831 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 05:04:21.468842 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c18a88efbe12492d93b70836c35441e0"
I20260924 05:04:21.468909 7101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:04:21.469112 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.469031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:38044)
0924 05:04:21.469068 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.469094 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:04:21.469168 7260 leader_election.cc:334] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
I20260924 05:04:21.469540 7101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd242b6afeff44eaa6c49cedc1eae3ea in term 3.
I20260924 05:04:21.469673 7263 leader_election.cc:302] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd242b6afeff44eaa6c49cedc1eae3ea, c18a88efbe12492d93b70836c35441e0; no voters: 1a98c13313924495b97777015844226a
I20260924 05:04:21.469791 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:04:21.469878 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [term 3 LEADER]: Becoming Leader. State: Replica: bd242b6afeff44eaa6c49cedc1eae3ea, State: Running, Role: LEADER
I20260924 05:04:21.469933 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } }
I20260924 05:04:21.470252 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd242b6afeff44eaa6c49cedc1eae3ea. Latest consensus state: current_term: 3 leader_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:21.470338 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:21.470494 7816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:21.470865 7816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:21.470928 7816 catalog_manager.cc:1259] Loaded cluster ID: dde3d3ee0e5f402eaa284404d019683e
I20260924 05:04:21.470958 7816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:21.471118 7816 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:21.471187 7816 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Loaded TSK: 0
I20260924 05:04:21.471319 7816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:21.501080 7276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:21.502650 7276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:21.503535 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.503442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:38070)
0924 05:04:21.503482 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.503507 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:04:21.503538 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [term 3 FOLLOWER]: Refusing update from remote peer bd242b6afeff44eaa6c49cedc1eae3ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 05:04:21.503719 7814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 05:04:21.503822 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.503738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.0.1:38086)
0924 05:04:21.503775 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.503801 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:04:21.503957 7260 consensus_peers.cc:487] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea -> Peer 1a98c13313924495b97777015844226a (127.0.98.62:42581): Couldn't send request to peer 1a98c13313924495b97777015844226a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 05:04:21.504279 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd242b6afeff44eaa6c49cedc1eae3ea. Latest consensus state: current_term: 3 leader_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:21.504362 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:04:21.504804 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:21.504879 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:21.505089 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1a98c13313924495b97777015844226a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42581 } } peers { permanent_uuid: "c18a88efbe12492d93b70836c35441e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45087 } attrs { promote: false } } peers { permanent_uuid: "bd242b6afeff44eaa6c49cedc1eae3ea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39645 } attrs { promote: false } } }
I20260924 05:04:21.505175 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18a88efbe12492d93b70836c35441e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:04:21.505234 7819 mvcc.cc:205] Tried to move back new op lower bound from 7332766767116804096 to 7332766766981345280. Current Snapshot: MvccSnapshot[applied={T|T < 7332766767116804096}]
I20260924 05:04:21.509647 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.506457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.0.98.3:36241)
0924 05:04:21.506734 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:21.506735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:21.506741 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:21.506760 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:21.506761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:21.506770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:21.506838 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:21.506963 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.507593 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.508783 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.509343 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.509414 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.509425 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:21.509535 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 05:04:21.509570 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":55,"threads_started":1}
I20260924 05:04:21.509651 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.506277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40019 (local address 127.0.0.1:56220)
0924 05:04:21.506305 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.506315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:21.506366 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:21.506688 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:21.506689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:21.506690 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:21.506728 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.506730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.507517 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.507518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.508646 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.508648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.509436 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.509437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.509498 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.509501 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:21.509529 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 05:04:21.509570 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":1}
I20260924 05:04:21.509738 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.506460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.98.2:36027)
0924 05:04:21.506908 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:21.506910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:21.507123 (+ 213us) server_negotiation.cc:407] Connection header received
0924 05:04:21.507150 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:21.507151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:21.507160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:21.507220 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:21.507352 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.507870 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.508717 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.509241 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.509299 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.509311 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:21.509604 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 05:04:21.509648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":49,"threads_started":1}
I20260924 05:04:21.509653 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.506355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36241 (local address 127.0.0.1:53430)
0924 05:04:21.506467 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.506476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:21.506521 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:21.506855 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:21.506856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:21.506857 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:21.506888 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.506890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.507676 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.507677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.508708 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.508710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.509421 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.509422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.509473 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.509475 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:21.509523 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:04:21.509556 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260924 05:04:21.509552 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.506356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36027 (local address 127.0.0.1:50860)
0924 05:04:21.507014 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.507023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:21.507078 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:21.507234 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:21.507235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:21.507237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:21.507290 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.507291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.507936 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.507938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.508642 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.508644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.509310 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.509311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.509381 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.509385 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:21.509430 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:04:21.509468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260924 05:04:21.509928 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.506476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.98.1:40019)
0924 05:04:21.506582 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:21.506584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:21.506591 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:21.506614 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:21.506615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:21.506624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:21.506673 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:21.506791 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.507454 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.508786 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.509346 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.509772 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.509789 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:21.509810 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 05:04:21.509855 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
I20260924 05:04:21.510093 7526 tablet_service.cc:1467] Processing CreateTablet for tablet e9b95ebd5e8d4873a860335be82e471e (DEFAULT_TABLE table=test-table [id=d42c32c3951c41bea0bc637c7b03dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:21.510143 7389 tablet_service.cc:1467] Processing CreateTablet for tablet e9b95ebd5e8d4873a860335be82e471e (DEFAULT_TABLE table=test-table [id=d42c32c3951c41bea0bc637c7b03dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:21.510222 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b95ebd5e8d4873a860335be82e471e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:21.510262 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b95ebd5e8d4873a860335be82e471e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:21.511286 7829 tablet_bootstrap.cc:492] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Bootstrap starting.
I20260924 05:04:21.511428 7662 tablet_service.cc:1467] Processing CreateTablet for tablet e9b95ebd5e8d4873a860335be82e471e (DEFAULT_TABLE table=test-table [id=d42c32c3951c41bea0bc637c7b03dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:21.511541 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b95ebd5e8d4873a860335be82e471e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:21.511660 7829 tablet_bootstrap.cc:654] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:21.512182 7829 tablet_bootstrap.cc:492] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: No bootstrap required, opened a new log
I20260924 05:04:21.512233 7829 ts_tablet_manager.cc:1397] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:21.512362 7829 raft_consensus.cc:348] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.512428 7829 raft_consensus.cc:374] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:21.512451 7829 raft_consensus.cc:729] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344cc91000b8475bad832b85c56a1bb7, State: Initialized, Role: FOLLOWER
I20260924 05:04:21.512497 7829 consensus_queue.cc:260] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.512688 7829 ts_tablet_manager.cc:1428] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:21.512935 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.512793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:58773)
0924 05:04:21.512887 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.512911 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 05:04:21.513087 7833 tablet_bootstrap.cc:492] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Bootstrap starting.
I20260924 05:04:21.513118 7592 heartbeater.cc:503] Master 127.0.98.60:39645 was elected leader, sending a full tablet report...
I20260924 05:04:21.513263 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.513187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:35761)
0924 05:04:21.513217 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.513241 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 05:04:21.513424 7833 tablet_bootstrap.cc:654] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:21.513656 7830 tablet_bootstrap.cc:492] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: Bootstrap starting.
I20260924 05:04:21.513947 7833 tablet_bootstrap.cc:492] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: No bootstrap required, opened a new log
I20260924 05:04:21.513979 7830 tablet_bootstrap.cc:654] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:21.513988 7833 ts_tablet_manager.cc:1397] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:21.514102 7833 raft_consensus.cc:348] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.514161 7833 raft_consensus.cc:374] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:21.514178 7833 raft_consensus.cc:729] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf97fa9703e45819ac3330ab207cac5, State: Initialized, Role: FOLLOWER
I20260924 05:04:21.514213 7833 consensus_queue.cc:260] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.514353 7833 ts_tablet_manager.cc:1428] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:21.514503 7830 tablet_bootstrap.cc:492] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: No bootstrap required, opened a new log
I20260924 05:04:21.514547 7830 ts_tablet_manager.cc:1397] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:21.514577 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.514497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:36911)
0924 05:04:21.514532 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.514553 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:04:21.514638 7830 raft_consensus.cc:348] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.514755 7830 raft_consensus.cc:374] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:21.514778 7830 raft_consensus.cc:729] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e7066d8cc64d23b9e155c02ae63c82, State: Initialized, Role: FOLLOWER
I20260924 05:04:21.514812 7728 heartbeater.cc:503] Master 127.0.98.60:39645 was elected leader, sending a full tablet report...
I20260924 05:04:21.514829 7830 consensus_queue.cc:260] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.514984 7830 ts_tablet_manager.cc:1428] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:21.515318 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.515231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:37511)
0924 05:04:21.515264 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.515293 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 05:04:21.515379 7455 heartbeater.cc:503] Master 127.0.98.60:39645 was elected leader, sending a full tablet report...
I20260924 05:04:21.515563 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.515490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:34441)
0924 05:04:21.515523 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.515544 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 05:04:21.549831 7458 tablet.cc:2369] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:04:21.558584 7731 tablet.cc:2369] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:21.619700 7836 raft_consensus.cc:482] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:21.619791 7836 raft_consensus.cc:504] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.620122 7836 leader_election.cc:288] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019), 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:21.623994 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.620372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36631 (local address 127.0.98.2:36027)
0924 05:04:21.620452 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:21.620454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:21.620618 (+ 164us) server_negotiation.cc:407] Connection header received
0924 05:04:21.620674 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:21.620675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:21.620685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:21.620751 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:21.620830 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.621516 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.623084 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.623653 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.623729 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.623742 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:21.623882 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:04:21.623918 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:21.623994 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.620344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45595 (local address 127.0.98.1:40019)
0924 05:04:21.620393 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:21.620395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:21.620419 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:04:21.620480 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:21.620481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:21.620493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:21.620551 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:21.620707 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.621509 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.623084 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.623650 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.623730 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.623742 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:21.623882 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:04:21.623923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:04:21.624015 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.620283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40019 (local address 127.0.98.3:45595)
0924 05:04:21.620344 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.620355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:21.620422 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:21.620559 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:21.620560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:21.620563 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:21.620644 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.620647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.621592 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.621593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.622972 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.622974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.623732 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.623733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.623814 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.623819 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:04:21.623870 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:04:21.623923 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":95}
I20260924 05:04:21.624025 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.620283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36027 (local address 127.0.98.3:36631)
0924 05:04:21.620556 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.620564 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:21.620611 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:21.620750 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:21.620750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:21.620752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:21.620791 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.620792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.621592 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.621593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.622970 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:21.622974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.623762 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.623763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:21.623817 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.623820 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:21.623870 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:04:21.623923 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":127,"thread_start_us":125,"threads_started":1}
I20260924 05:04:21.624276 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b95ebd5e8d4873a860335be82e471e" candidate_uuid: "9cf97fa9703e45819ac3330ab207cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e7066d8cc64d23b9e155c02ae63c82" is_pre_election: true
I20260924 05:04:21.624406 7409 raft_consensus.cc:2454] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf97fa9703e45819ac3330ab207cac5 in term 0.
I20260924 05:04:21.624594 7618 leader_election.cc:302] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e7066d8cc64d23b9e155c02ae63c82, 9cf97fa9703e45819ac3330ab207cac5; no voters:
I20260924 05:04:21.624650 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b95ebd5e8d4873a860335be82e471e" candidate_uuid: "9cf97fa9703e45819ac3330ab207cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344cc91000b8475bad832b85c56a1bb7" is_pre_election: true
I20260924 05:04:21.624740 7546 raft_consensus.cc:2454] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf97fa9703e45819ac3330ab207cac5 in term 0.
I20260924 05:04:21.624850 7836 raft_consensus.cc:2781] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:21.624898 7836 raft_consensus.cc:482] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:21.624924 7836 raft_consensus.cc:3037] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:21.625574 7836 raft_consensus.cc:504] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.625702 7836 leader_election.cc:288] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [CANDIDATE]: Term 1 election: Requested vote from peers 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019), 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027)
I20260924 05:04:21.625846 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b95ebd5e8d4873a860335be82e471e" candidate_uuid: "9cf97fa9703e45819ac3330ab207cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e7066d8cc64d23b9e155c02ae63c82"
I20260924 05:04:21.625869 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b95ebd5e8d4873a860335be82e471e" candidate_uuid: "9cf97fa9703e45819ac3330ab207cac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "344cc91000b8475bad832b85c56a1bb7"
I20260924 05:04:21.625928 7546 raft_consensus.cc:3037] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:21.625936 7409 raft_consensus.cc:3037] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:21.626667 7546 raft_consensus.cc:2454] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf97fa9703e45819ac3330ab207cac5 in term 1.
I20260924 05:04:21.626667 7409 raft_consensus.cc:2454] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf97fa9703e45819ac3330ab207cac5 in term 1.
I20260924 05:04:21.626811 7615 leader_election.cc:302] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 344cc91000b8475bad832b85c56a1bb7, 9cf97fa9703e45819ac3330ab207cac5; no voters:
I20260924 05:04:21.626924 7836 raft_consensus.cc:2781] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:21.627102 7836 raft_consensus.cc:686] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf97fa9703e45819ac3330ab207cac5, State: Running, Role: LEADER
I20260924 05:04:21.627169 7836 consensus_queue.cc:237] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } }
I20260924 05:04:21.627451 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.627341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:60159)
0924 05:04:21.627394 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.627427 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 05:04:21.627885 7276 catalog_manager.cc:5495] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9cf97fa9703e45819ac3330ab207cac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf97fa9703e45819ac3330ab207cac5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:21.653746 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.651844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.3:36241)
0924 05:04:21.651887 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:21.651888 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:21.651897 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:21.651931 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:21.651932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:21.651949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:21.652002 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:21.652092 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.652541 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.653179 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:21.653349 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:21.653389 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:21.653480 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:21.653570 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:21.653622 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:21.653638 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 05:04:21.653652 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:04:21.654734 7546 raft_consensus.cc:1261] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Refusing update from remote peer 9cf97fa9703e45819ac3330ab207cac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:21.654752 7409 raft_consensus.cc:1261] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Refusing update from remote peer 9cf97fa9703e45819ac3330ab207cac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:21.654965 7836 consensus_queue.cc:1035] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "344cc91000b8475bad832b85c56a1bb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:21.655103 7839 consensus_queue.cc:1035] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e7066d8cc64d23b9e155c02ae63c82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:21.655957 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.655812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:58269)
0924 05:04:21.655895 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.655924 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 05:04:21.656234 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.656013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:49045)
0924 05:04:21.656171 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.656205 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 05:04:21.657431 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.657298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.1:44027)
0924 05:04:21.657375 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.657406 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 05:04:21.657748 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.657641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.3:39105)
0924 05:04:21.657676 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.657700 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:04:21.658298 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.658207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42581 (local address 127.0.98.2:52689)
0924 05:04:21.658247 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.658270 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:04:21.659103 7268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:21.659195 7268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:21.660239 7268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd242b6afeff44eaa6c49cedc1eae3ea: Sending DeleteTablet for 3 replicas of tablet e9b95ebd5e8d4873a860335be82e471e
I20260924 05:04:21.660481 7526 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b95ebd5e8d4873a860335be82e471e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:21 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50860
I20260924 05:04:21.660665 7852 tablet_replica.cc:330] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: stopping tablet replica
I20260924 05:04:21.660718 7852 raft_consensus.cc:2229] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:21.660708 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b95ebd5e8d4873a860335be82e471e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:21 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53430
I20260924 05:04:21.660763 7852 raft_consensus.cc:2258] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:21.660876 7853 tablet_replica.cc:330] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: stopping tablet replica
I20260924 05:04:21.660903 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b95ebd5e8d4873a860335be82e471e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:21 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56220
I20260924 05:04:21.660926 7853 raft_consensus.cc:2229] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:21.660997 7853 raft_consensus.cc:2258] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:21.661090 7854 tablet_replica.cc:330] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: stopping tablet replica
I20260924 05:04:21.661141 7854 raft_consensus.cc:2229] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:21.661183 7854 raft_consensus.cc:2258] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:21.661358 7853 ts_tablet_manager.cc:1905] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:21.661356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 7238
I20260924 05:04:21.661419 7854 ts_tablet_manager.cc:1905] T e9b95ebd5e8d4873a860335be82e471e P 87e7066d8cc64d23b9e155c02ae63c82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:21.662894 7853 ts_tablet_manager.cc:1918] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:21.662941 7853 log.cc:1196] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/e9b95ebd5e8d4873a860335be82e471e
I20260924 05:04:21.663146 7853 ts_tablet_manager.cc:1939] T e9b95ebd5e8d4873a860335be82e471e P 9cf97fa9703e45819ac3330ab207cac5: Deleting consensus metadata
I20260924 05:04:21.663404 7263 catalog_manager.cc:4841] TS 9cf97fa9703e45819ac3330ab207cac5 (127.0.98.3:36241): tablet e9b95ebd5e8d4873a860335be82e471e (table test-table [id=d42c32c3951c41bea0bc637c7b03dd60]) successfully deleted
I20260924 05:04:21.665957 7852 ts_tablet_manager.cc:1905] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:21.667160 7852 ts_tablet_manager.cc:1918] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:21.667205 7852 log.cc:1196] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/e9b95ebd5e8d4873a860335be82e471e
I20260924 05:04:21.667385 7852 ts_tablet_manager.cc:1939] T e9b95ebd5e8d4873a860335be82e471e P 344cc91000b8475bad832b85c56a1bb7: Deleting consensus metadata
I20260924 05:04:21.667627 7260 catalog_manager.cc:4841] TS 344cc91000b8475bad832b85c56a1bb7 (127.0.98.2:36027): tablet e9b95ebd5e8d4873a860335be82e471e (table test-table [id=d42c32c3951c41bea0bc637c7b03dd60]) successfully deleted
I20260924 05:04:21.669653 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:21.669558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40019 (local address 127.0.0.1:56228)
0924 05:04:21.669602 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:21.669626 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:04:21.669667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 7463
W20260924 05:04:21.669786 7263 catalog_manager.cc:4584] TS 87e7066d8cc64d23b9e155c02ae63c82 (127.0.98.1:40019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9b95ebd5e8d4873a860335be82e471e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40019: connect: Connection refused (error 111)
I20260924 05:04:21.675391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 7599
I20260924 05:04:21.679627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 6955
I20260924 05:04:21.683326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 7145
2026-09-24T05:04:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5852 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:21 dist-test-slave-m01t krb5kdc[7857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:21 dist-test-slave-m01t krb5kdc[7857](info): set up 2 sockets
Sep 24 05:04:21 dist-test-slave-m01t krb5kdc[7857](info): commencing operation
krb5kdc: starting...
W20260924 05:04:22.629823 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:22 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 05:04:22.732393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:22.809297 7870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:22.809484 7870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:22.809504 7870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:22.810582 7870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:22.810612 7870 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 05:04:22.810623 7870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:22.810635 7870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:22.810645 7870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:22.810658 7870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:22.811957 7870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:22.812124 7870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:22.812273 7870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:22.816445 7876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:22.816502 7870 server_base.cc:1034] running on GCE node
W20260924 05:04:22.816474 7875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:22.816468 7878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:22.816905 7870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:22.817158 7870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:22.818290 7870 hybrid_clock.cc:648] HybridClock initialized: now 1790226262818274 us; error 32 us; skew 500 ppm
Sep 24 05:04:22 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:22.826190 7870 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 05:04:22.826632 7870 webserver.cc:460] Webserver started at http://127.0.98.62:41233/ using document root <none> and password file <none>
I20260924 05:04:22.826834 7870 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:22.826882 7870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:22.827013 7870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:22.828034 7870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "a2896f40bb0940c2be9fbff58083d9b0"
format_stamp: "Formatted at 2026-09-24 05:04:22 on dist-test-slave-m01t"
server_key: "a1360008049a5f21ee79008f8efd36b7"
server_key_iv: "60c7489df7735c74be35b0cecc8c6247"
server_key_version: "encryptionkey@0"
I20260924 05:04:22.828356 7870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2896f40bb0940c2be9fbff58083d9b0"
format_stamp: "Formatted at 2026-09-24 05:04:22 on dist-test-slave-m01t"
server_key: "a1360008049a5f21ee79008f8efd36b7"
server_key_iv: "60c7489df7735c74be35b0cecc8c6247"
server_key_version: "encryptionkey@0"
I20260924 05:04:22.829583 7870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:22.830713 7885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:22.831239 7870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:22.831331 7870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "a2896f40bb0940c2be9fbff58083d9b0"
format_stamp: "Formatted at 2026-09-24 05:04:22 on dist-test-slave-m01t"
server_key: "a1360008049a5f21ee79008f8efd36b7"
server_key_iv: "60c7489df7735c74be35b0cecc8c6247"
server_key_version: "encryptionkey@0"
I20260924 05:04:22.831404 7870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:22.849189 7870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:22.851555 7870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:22.851688 7870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:22.856554 7870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35511
I20260924 05:04:22.856618 7947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35511 every 8 connection(s)
I20260924 05:04:22.856945 7870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:22.857652 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:22.860771 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Bootstrap starting.
I20260924 05:04:22.861461 7948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:22.861780 7948 log.cc:824] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:22.862516 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: No bootstrap required, opened a new log
I20260924 05:04:22.865331 7948 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2896f40bb0940c2be9fbff58083d9b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:04:22.865506 7948 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:22.865548 7948 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2896f40bb0940c2be9fbff58083d9b0, State: Initialized, Role: FOLLOWER
I20260924 05:04:22.865696 7948 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2896f40bb0940c2be9fbff58083d9b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:04:22.865816 7948 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:22.865857 7948 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:22.865901 7948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:22.866336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 7870
I20260924 05:04:22.866421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:22.866545 7948 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2896f40bb0940c2be9fbff58083d9b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:04:22.866606 392 external_mini_cluster.cc:1428] Setting key 8b1c2a222eb0750bc4532aa5a4d71c9d
I20260924 05:04:22.866647 7948 leader_election.cc:302] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2896f40bb0940c2be9fbff58083d9b0; no voters:
I20260924 05:04:22.866801 7948 leader_election.cc:288] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:22.866827 7951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:22.866935 7951 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [term 1 LEADER]: Becoming Leader. State: Replica: a2896f40bb0940c2be9fbff58083d9b0, State: Running, Role: LEADER
I20260924 05:04:22.867038 7948 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:22.867017 7951 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2896f40bb0940c2be9fbff58083d9b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:04:22.867363 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2896f40bb0940c2be9fbff58083d9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2896f40bb0940c2be9fbff58083d9b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } } }
I20260924 05:04:22.867451 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:22.867367 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2896f40bb0940c2be9fbff58083d9b0. Latest consensus state: current_term: 1 leader_uuid: "a2896f40bb0940c2be9fbff58083d9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2896f40bb0940c2be9fbff58083d9b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } } }
I20260924 05:04:22.867894 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:22.868247 7959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:22.868703 7959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:22.871701 7959 catalog_manager.cc:1347] Generated new cluster ID: 572e166e9ff343c1b8cb19b945668697
I20260924 05:04:22.871749 7959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:22 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:22.878829 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:22.867173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.98.62:35511)
0924 05:04:22.869295 (+ 2122us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:22.869304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:22.869333 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:04:22.869445 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:22.869452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:22.869504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:22.870336 (+ 832us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:22.870402 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:22.871261 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:22.872357 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:22.872956 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:22.873021 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:22.875581 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:22.875585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:22.875585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:22.875596 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:22.877240 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:22.877581 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:22.877584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:22.877585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:22.877681 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:22.877871 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:22.877871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:22.877872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:22.878175 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:22.878283 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:22.878507 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 05:04:22.878627 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2075,"thread_start_us":61,"threads_started":1}
I20260924 05:04:22.885667 7959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:22.886301 7959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:22.906013 7959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Generated new TSK 0
I20260924 05:04:22.906214 7959 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 05:04:22.932085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35511
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:23.009217 7974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:23.009397 7974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:23.009418 7974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:23.010499 7974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:23.010536 7974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:23.010555 7974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:04:23.010573 7974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:23.011929 7974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:23.012095 7974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:23.012253 7974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:23.016599 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:23.016623 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:23.016652 7974 server_base.cc:1034] running on GCE node
W20260924 05:04:23.016623 7982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:23.016939 7974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:23.017161 7974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:23.018321 7974 hybrid_clock.cc:648] HybridClock initialized: now 1790226263018303 us; error 30 us; skew 500 ppm
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:23.020511 7974 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:04:23.020867 7974 webserver.cc:460] Webserver started at http://127.0.98.1:32811/ using document root <none> and password file <none>
I20260924 05:04:23.021032 7974 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:23.021070 7974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:23.021163 7974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:23.022150 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "43f6f3116f4a4908a45ad65f72a89bd1"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "abe6e3a260a9629c477ebdd8a0077954"
server_key_iv: "86f8b2174ec45d3d6264f19feedcc534"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.022432 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "43f6f3116f4a4908a45ad65f72a89bd1"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "abe6e3a260a9629c477ebdd8a0077954"
server_key_iv: "86f8b2174ec45d3d6264f19feedcc534"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.023535 7974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:23.024130 7989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.024273 7974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:23.024350 7974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "43f6f3116f4a4908a45ad65f72a89bd1"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "abe6e3a260a9629c477ebdd8a0077954"
server_key_iv: "86f8b2174ec45d3d6264f19feedcc534"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.024415 7974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:23.037371 7974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:23.039466 7974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:23.039602 7974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:23.040076 7974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:23.040123 7974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.040154 7974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:23.040179 7974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.045953 7974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42955
I20260924 05:04:23.046090 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42955 every 8 connection(s)
I20260924 05:04:23.046327 7974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:23.049026 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.040545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46329 (local address 127.0.98.62:35511)
0924 05:04:23.040608 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.040611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.040777 (+ 166us) server_negotiation.cc:407] Connection header received
0924 05:04:23.041301 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.041302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.041319 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.041387 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.042220 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.042965 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.044382 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.044689 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.044764 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.046997 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.046998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.046999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.047006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.048049 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.048363 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.048363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.048364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.048404 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.048568 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.048568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.048568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.048665 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.048708 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.048897 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.048964 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:04:23.049152 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.040521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.98.1:46329)
0924 05:04:23.040670 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.040709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.041187 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.041478 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.041483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.041848 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.042123 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.042129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.043054 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.043055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.044293 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.044295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.044762 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.044763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.044902 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.045385 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.045395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.046918 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.048108 (+ 1190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.048114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.048115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.048309 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.048444 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.048445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.048445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.048528 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.048703 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.048706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.048809 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.048880 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.048985 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260924 05:04:23.049612 8105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35511
I20260924 05:04:23.049687 8105 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:23.049877 8105 heartbeater.cc:511] Master 127.0.98.62:35511 requested a full tablet report, sending...
I20260924 05:04:23.050360 7902 ts_manager.cc:194] Registered new tserver with Master: 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955)
I20260924 05:04:23.051193 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46329
I20260924 05:04:23.054922 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.050625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55153 (local address 127.0.98.62:35511)
0924 05:04:23.050660 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.050662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.050723 (+ 61us) server_negotiation.cc:407] Connection header received
0924 05:04:23.050784 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.050785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.050792 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.050832 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.051109 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.051662 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.052321 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.052498 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.052548 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.053188 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.053189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.053190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.053194 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.054221 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.054471 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.054471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.054471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.054510 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.054623 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.054623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.054624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.054701 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.054739 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.054820 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.054871 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:23.054929 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.050559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.98.1:55153)
0924 05:04:23.050671 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.050681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.050736 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.050836 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.050836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.051006 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.051059 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.051060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.051755 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.051755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.052263 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.052265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.052556 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.052557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.052596 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.052891 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.052895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.053129 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.054282 (+ 1153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.054282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.054283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.054421 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.054545 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.054545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.054546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.054584 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.054733 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.054733 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.054761 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.054805 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.054866 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
I20260924 05:04:23.056366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 7974
I20260924 05:04:23.056453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:23.056643 392 external_mini_cluster.cc:1428] Setting key 81ccc9884a8348b66d5497f28a2d537e
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 05:04:23.076507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35511
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:23.153647 8114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:23.153877 8114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:23.153900 8114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:23.155028 8114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:23.155066 8114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:23.155085 8114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:04:23.155114 8114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:23.156530 8114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:23.156724 8114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:23.156913 8114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:23.161269 8120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:23.161309 8122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:23.161418 8114 server_base.cc:1034] running on GCE node
W20260924 05:04:23.161307 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:23.161656 8114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:23.161898 8114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:23.163038 8114 hybrid_clock.cc:648] HybridClock initialized: now 1790226263163024 us; error 26 us; skew 500 ppm
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:23.165174 8114 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:04:23.165560 8114 webserver.cc:460] Webserver started at http://127.0.98.2:39805/ using document root <none> and password file <none>
I20260924 05:04:23.165779 8114 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:23.165827 8114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:23.165946 8114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:23.166880 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b753695418240298a35204a7db5f55c"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "170c9d3d2ccb1ffa5e465ea0d3036257"
server_key_iv: "22b3c6fea0edc433be7b84601f400a4a"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.167239 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b753695418240298a35204a7db5f55c"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "170c9d3d2ccb1ffa5e465ea0d3036257"
server_key_iv: "22b3c6fea0edc433be7b84601f400a4a"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.168442 8114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:23.169064 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.169209 8114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:23.169284 8114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "3b753695418240298a35204a7db5f55c"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "170c9d3d2ccb1ffa5e465ea0d3036257"
server_key_iv: "22b3c6fea0edc433be7b84601f400a4a"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.169342 8114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:23.180822 8114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:23.182345 8114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:23.182462 8114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:23.182897 8114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:23.182934 8114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.182961 8114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:23.182984 8114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:23.189697 8114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33793
I20260924 05:04:23.189780 8244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33793 every 8 connection(s)
I20260924 05:04:23.190155 8114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:23.191707 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.183203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41415 (local address 127.0.98.62:35511)
0924 05:04:23.183260 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.183262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.183471 (+ 209us) server_negotiation.cc:407] Connection header received
0924 05:04:23.184049 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.184050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.184059 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.184096 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.184855 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.185339 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.186789 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.186987 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.187043 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.189257 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.189259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.189259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.189266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.190727 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.191048 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.191049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.191049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.191096 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.191263 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.191264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.191264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.191358 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.191401 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.191587 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.191646 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:04:23.191865 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.183188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.98.2:41415)
0924 05:04:23.183339 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.183382 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.183949 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.184169 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.184173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.184479 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.184759 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.184765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.185411 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.185411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.186707 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.186711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.187056 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.187057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.187193 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.187663 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.187673 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.189165 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.190790 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.190793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.190795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.190991 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.191139 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.191140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.191140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.191222 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.191397 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.191400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.191502 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.191573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.191691 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260924 05:04:23.192246 8245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35511
I20260924 05:04:23.192317 8245 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:23.192464 8245 heartbeater.cc:511] Master 127.0.98.62:35511 requested a full tablet report, sending...
I20260924 05:04:23.192813 7893 ts_manager.cc:194] Registered new tserver with Master: 3b753695418240298a35204a7db5f55c (127.0.98.2:33793)
I20260924 05:04:23.193225 7893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41415
I20260924 05:04:23.197357 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.193186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56553 (local address 127.0.98.62:35511)
0924 05:04:23.193222 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.193224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.193269 (+ 45us) server_negotiation.cc:407] Connection header received
0924 05:04:23.193350 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.193351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.193360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.193413 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.193741 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.194210 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.194822 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.194987 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.195034 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.195672 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.195673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.195674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.195679 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.196616 (+ 937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.196866 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.196866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.196866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.196906 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.197007 (+ 101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.197007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.197007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.197123 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.197167 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.197254 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.197303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:23.197363 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.193104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.98.2:56553)
0924 05:04:23.193214 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.193224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.193284 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.193419 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.193419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.193582 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.193652 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.193654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.194287 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.194288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.194768 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.194769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.195032 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.195032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.195076 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.195376 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.195380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.195608 (+ 228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.196676 (+ 1068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.196677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.196677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.196815 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.196938 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.196938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.196939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.196977 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.197165 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.197166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.197194 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.197237 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.197300 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260924 05:04:23.200145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 8114
I20260924 05:04:23.200234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:23.200383 392 external_mini_cluster.cc:1428] Setting key 3d26b71706e135d0746c748af929487d
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 05:04:23.220019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35511
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:23.296334 8254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:23.296514 8254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:23.296536 8254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:23.297673 8254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:23.297729 8254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:23.297765 8254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:04:23.297793 8254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:23.299186 8254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:23.299384 8254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:23.299571 8254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:23.303953 8259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:23.303934 8262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:23.303939 8260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:23.304242 8254 server_base.cc:1034] running on GCE node
I20260924 05:04:23.304415 8254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:23.304589 8254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:23.305733 8254 hybrid_clock.cc:648] HybridClock initialized: now 1790226263305719 us; error 27 us; skew 500 ppm
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:23.307934 8254 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:04:23.308292 8254 webserver.cc:460] Webserver started at http://127.0.98.3:39581/ using document root <none> and password file <none>
I20260924 05:04:23.308465 8254 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:23.308504 8254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:23.308619 8254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:23.309522 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "1df1ca6611a449f13992507524aaf781"
server_key_iv: "ea0815cf3a9b1a1aeaecbf2f2e8427d7"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.309855 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "1df1ca6611a449f13992507524aaf781"
server_key_iv: "ea0815cf3a9b1a1aeaecbf2f2e8427d7"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.310951 8254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:04:23.311523 8269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.311650 8254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:23.311717 8254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
format_stamp: "Formatted at 2026-09-24 05:04:23 on dist-test-slave-m01t"
server_key: "1df1ca6611a449f13992507524aaf781"
server_key_iv: "ea0815cf3a9b1a1aeaecbf2f2e8427d7"
server_key_version: "encryptionkey@0"
I20260924 05:04:23.311775 8254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:23.325148 8254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:23.327347 8254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:23.327471 8254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:23.327946 8254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:23.327983 8254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.328003 8254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:23.328016 8254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.333921 8254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40615
I20260924 05:04:23.333990 8384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40615 every 8 connection(s)
I20260924 05:04:23.334232 8254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:23.336952 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.328360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43431 (local address 127.0.98.62:35511)
0924 05:04:23.328419 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.328422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.328620 (+ 198us) server_negotiation.cc:407] Connection header received
0924 05:04:23.329102 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.329103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.329123 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.329194 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.329954 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.330711 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.332142 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.332436 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.332517 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.334811 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.334812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.334812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.334819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.335883 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.336230 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.336230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.336231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.336277 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.336452 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.336452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.336452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.336564 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.336619 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.336830 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.336888 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:04:23.337128 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.328334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.98.3:43431)
0924 05:04:23.328496 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.328538 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.329009 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.329267 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.329271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.329586 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.329862 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.329868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.330796 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.330797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.332050 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.332053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.332515 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.332516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.332616 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.333107 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.333116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.334729 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.335949 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.335954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.335956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.336168 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.336323 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.336323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.336324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.336410 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.336614 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.336618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.336725 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.336813 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.336925 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260924 05:04:23.337531 8385 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35511
I20260924 05:04:23.337605 8385 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:23.337805 8385 heartbeater.cc:511] Master 127.0.98.62:35511 requested a full tablet report, sending...
I20260924 05:04:23.338166 7902 ts_manager.cc:194] Registered new tserver with Master: cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.338619 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43431
I20260924 05:04:23.342999 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.338524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34835 (local address 127.0.98.62:35511)
0924 05:04:23.338562 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.338563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.338621 (+ 58us) server_negotiation.cc:407] Connection header received
0924 05:04:23.338681 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.338682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.338691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.338733 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.339057 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.339627 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.340239 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.340417 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.340456 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.341117 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.341118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.341119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.341124 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.342299 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.342541 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.342542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.342542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.342582 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.342693 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.342693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.342693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.342771 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.342811 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.342893 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.342944 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:23.343008 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.338460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.98.3:34835)
0924 05:04:23.338567 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.338577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.338631 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.338749 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.338749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.338933 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.338992 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.338993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.339697 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.339697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.340189 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.340190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.340461 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.340461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.340502 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.340818 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.340822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.341057 (+ 235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.342353 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.342354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.342354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.342488 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.342621 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.342621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.342622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.342661 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.342807 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.342808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.342836 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.342878 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.342943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260924 05:04:23.343734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 8254
I20260924 05:04:23.343839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:23.344045 392 external_mini_cluster.cc:1428] Setting key 37dbe04c3b8e63db13b87a5f0e80ddab
I20260924 05:04:23.344712 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:04:23.363093 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:23.358658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.0.98.62:35511)
0924 05:04:23.358696 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.358698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.358708 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:23.358758 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.358759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.358766 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.358809 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.359280 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.359744 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.360384 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.360558 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.360597 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.363021 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41480: BlockingRecv error: recv got EOF from 127.0.0.1:41480 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:23.369275 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.363708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.98.62:35511)
0924 05:04:23.363756 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.363758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.363779 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:04:23.363827 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.363827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.363834 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.363875 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.364252 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.364676 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.365245 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.365404 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.365441 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.367233 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.367234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.367234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.367240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.368393 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.368773 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.368774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.368774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.368819 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.368932 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.368932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.368933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.369015 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.369086 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.369161 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.369211 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:04:23.370580 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:23.372102 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:23.377962 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.373966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42955 (local address 127.0.0.1:53822)
0924 05:04:23.374457 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.374464 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.374503 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.374656 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.374657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.374659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.374701 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.374702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.375621 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.375622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.376831 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.376833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.377672 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.377765 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.377782 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.377849 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.377896 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":36,"threads_started":1}
I20260924 05:04:23.378001 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.374171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.0.98.1:42955)
0924 05:04:23.374333 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.374336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.374508 (+ 172us) server_negotiation.cc:407] Connection header received
0924 05:04:23.374573 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.374576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.374589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.374654 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.374823 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.375547 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.376896 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.377592 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377662 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.377755 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.377872 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.377909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260924 05:04:23.378201 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.373949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40615 (local address 127.0.0.1:52216)
0924 05:04:23.374269 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.374311 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.374374 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.374610 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.374613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.374616 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.374671 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.374672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.375901 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.375902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.376936 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.376938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.377850 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.377925 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.377930 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.377979 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.378139 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":47,"threads_started":1}
I20260924 05:04:23.378482 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.374100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.0.98.3:40615)
0924 05:04:23.374260 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.374264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.374378 (+ 114us) server_negotiation.cc:407] Connection header received
0924 05:04:23.374444 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.374446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.374460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.374540 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.374747 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.375823 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377004 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.377676 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377754 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.377817 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.378348 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.378393 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260924 05:04:23.378660 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcd38163a6f4108ae4ed27893c19115 (DEFAULT_TABLE table=test-table [id=518e80d46f5d45e79d49c0c8d94d5b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.378842 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.373952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33793 (local address 127.0.0.1:57800)
0924 05:04:23.374295 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.374311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.374375 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.374622 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.374623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.374624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.374676 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.374677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.377005 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.377651 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.377653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.378628 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.378629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.378688 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.378691 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.378747 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.378781 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":4,"thread_start_us":40,"threads_started":1}
I20260924 05:04:23.378895 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.374155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.98.2:33793)
0924 05:04:23.374393 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.374397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.374405 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:23.374444 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.374446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.374460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.374540 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.374746 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.375827 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.377760 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.378465 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.378527 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.378596 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.378769 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.378804 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
I20260924 05:04:23.379026 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcd38163a6f4108ae4ed27893c19115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.379437 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcd38163a6f4108ae4ed27893c19115 (DEFAULT_TABLE table=test-table [id=518e80d46f5d45e79d49c0c8d94d5b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.379704 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcd38163a6f4108ae4ed27893c19115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.382089 8410 tablet_bootstrap.cc:492] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Bootstrap starting.
I20260924 05:04:23.382557 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcd38163a6f4108ae4ed27893c19115 (DEFAULT_TABLE table=test-table [id=518e80d46f5d45e79d49c0c8d94d5b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.382675 8410 tablet_bootstrap.cc:654] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.382876 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcd38163a6f4108ae4ed27893c19115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.382975 8410 log.cc:824] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:23.383682 8410 tablet_bootstrap.cc:492] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: No bootstrap required, opened a new log
I20260924 05:04:23.383742 8410 ts_tablet_manager.cc:1397] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.385584 8410 raft_consensus.cc:348] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.385776 8410 raft_consensus.cc:374] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.385813 8410 raft_consensus.cc:729] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.385947 8410 consensus_queue.cc:260] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.386237 8105 heartbeater.cc:503] Master 127.0.98.62:35511 was elected leader, sending a full tablet report...
I20260924 05:04:23.386216 8410 ts_tablet_manager.cc:1428] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:04:23.387411 8414 tablet_bootstrap.cc:492] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Bootstrap starting.
I20260924 05:04:23.387594 8411 tablet_bootstrap.cc:492] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Bootstrap starting.
I20260924 05:04:23.387948 8414 tablet_bootstrap.cc:654] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.388091 8411 tablet_bootstrap.cc:654] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.388206 8414 log.cc:824] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:23.388355 8411 log.cc:824] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:23.388876 8414 tablet_bootstrap.cc:492] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: No bootstrap required, opened a new log
I20260924 05:04:23.388932 8414 ts_tablet_manager.cc:1397] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.389002 8411 tablet_bootstrap.cc:492] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: No bootstrap required, opened a new log
I20260924 05:04:23.389063 8411 ts_tablet_manager.cc:1397] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.390334 8414 raft_consensus.cc:348] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.390491 8414 raft_consensus.cc:374] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.390532 8414 raft_consensus.cc:729] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2f6b4069ac498f92f3ebe6e5eaed11, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.390456 8411 raft_consensus.cc:348] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.390618 8411 raft_consensus.cc:374] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.390657 8411 raft_consensus.cc:729] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b753695418240298a35204a7db5f55c, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.390671 8414 consensus_queue.cc:260] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.390763 8411 consensus_queue.cc:260] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.390910 8414 ts_tablet_manager.cc:1428] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:23.390940 8385 heartbeater.cc:503] Master 127.0.98.62:35511 was elected leader, sending a full tablet report...
I20260924 05:04:23.390995 8411 ts_tablet_manager.cc:1428] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:23.391072 8245 heartbeater.cc:503] Master 127.0.98.62:35511 was elected leader, sending a full tablet report...
W20260924 05:04:23.440704 8246 tablet.cc:2369] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:23.504237 8418 raft_consensus.cc:482] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:23.504347 8418 raft_consensus.cc:504] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.504650 8418 leader_election.cc:288] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615), 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955)
I20260924 05:04:23.507359 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.504814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42955 (local address 127.0.98.2:52069)
0924 05:04:23.504860 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.504872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.504915 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.505055 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.505056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.505058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.505127 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.505128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.505736 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.505737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.506610 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.506613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.507119 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.507120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.507191 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.507194 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.507239 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.507275 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":42}
I20260924 05:04:23.507347 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.504887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52069 (local address 127.0.98.1:42955)
0924 05:04:23.504929 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.504930 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.504940 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:23.504968 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.504968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.504980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.505039 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.505191 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.505649 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.506694 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.507044 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.507101 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.507112 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.507254 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.507287 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:23.507445 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.504935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51723 (local address 127.0.98.3:40615)
0924 05:04:23.504971 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.504973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.505099 (+ 126us) server_negotiation.cc:407] Connection header received
0924 05:04:23.505130 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.505131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.505142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.505187 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.505310 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.505809 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.506789 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.507160 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.507217 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.507226 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.507352 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.507384 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:23.507632 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.504819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40615 (local address 127.0.98.2:51723)
0924 05:04:23.505039 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.505046 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.505089 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.505194 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.505194 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.505196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.505258 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.505259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.505867 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.505868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.506713 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.506716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.507225 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.507226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.507295 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.507299 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.507341 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.507379 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260924 05:04:23.507658 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcd38163a6f4108ae4ed27893c19115" candidate_uuid: "3b753695418240298a35204a7db5f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" is_pre_election: true
I20260924 05:04:23.507790 8059 raft_consensus.cc:2454] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b753695418240298a35204a7db5f55c in term 0.
I20260924 05:04:23.507944 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcd38163a6f4108ae4ed27893c19115" candidate_uuid: "3b753695418240298a35204a7db5f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" is_pre_election: true
I20260924 05:04:23.507987 8132 leader_election.cc:302] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b753695418240298a35204a7db5f55c, 43f6f3116f4a4908a45ad65f72a89bd1; no voters:
I20260924 05:04:23.508075 8339 raft_consensus.cc:2454] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b753695418240298a35204a7db5f55c in term 0.
I20260924 05:04:23.508096 8418 raft_consensus.cc:2781] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:23.508134 8418 raft_consensus.cc:482] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:23.508160 8418 raft_consensus.cc:3037] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.509058 8418 raft_consensus.cc:504] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.509168 8418 leader_election.cc:288] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [CANDIDATE]: Term 1 election: Requested vote from peers cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615), 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955)
I20260924 05:04:23.509297 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcd38163a6f4108ae4ed27893c19115" candidate_uuid: "3b753695418240298a35204a7db5f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f6f3116f4a4908a45ad65f72a89bd1"
I20260924 05:04:23.509364 8059 raft_consensus.cc:3037] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.510033 8059 raft_consensus.cc:2454] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b753695418240298a35204a7db5f55c in term 1.
I20260924 05:04:23.510135 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcd38163a6f4108ae4ed27893c19115" candidate_uuid: "3b753695418240298a35204a7db5f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
I20260924 05:04:23.510191 8132 leader_election.cc:302] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b753695418240298a35204a7db5f55c, 43f6f3116f4a4908a45ad65f72a89bd1; no voters:
I20260924 05:04:23.510210 8339 raft_consensus.cc:3037] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.510258 8418 raft_consensus.cc:2781] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:23.510381 8418 raft_consensus.cc:686] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 1 LEADER]: Becoming Leader. State: Replica: 3b753695418240298a35204a7db5f55c, State: Running, Role: LEADER
I20260924 05:04:23.510462 8418 consensus_queue.cc:237] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } }
I20260924 05:04:23.510885 8339 raft_consensus.cc:2454] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b753695418240298a35204a7db5f55c in term 1.
I20260924 05:04:23.511356 7893 catalog_manager.cc:5495] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b753695418240298a35204a7db5f55c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3b753695418240298a35204a7db5f55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:23.520418 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.518561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.98.2:33793)
0924 05:04:23.518594 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.518596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.518602 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:23.518657 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.518657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.518671 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.518725 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:23.518810 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.519222 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.519869 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.520052 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.520094 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.520198 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:23.520279 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:23.520328 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.520345 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.520360 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:23.521993 8059 raft_consensus.cc:1261] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Refusing update from remote peer 3b753695418240298a35204a7db5f55c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:23.522047 8339 raft_consensus.cc:1261] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Refusing update from remote peer 3b753695418240298a35204a7db5f55c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:23.522285 8420 consensus_queue.cc:1035] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:23.522390 8418 consensus_queue.cc:1035] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:23.524361 8423 mvcc.cc:205] Tried to move back new op lower bound from 7332766775383588864 to 7332766775339360256. Current Snapshot: MvccSnapshot[applied={T|T < 7332766775383588864}]
I20260924 05:04:23.524585 8425 mvcc.cc:205] Tried to move back new op lower bound from 7332766775383588864 to 7332766775339360256. Current Snapshot: MvccSnapshot[applied={T|T < 7332766775383588864}]
I20260924 05:04:23.524785 8424 mvcc.cc:205] Tried to move back new op lower bound from 7332766775383588864 to 7332766775339360256. Current Snapshot: MvccSnapshot[applied={T|T < 7332766775383588864}]
I20260924 05:04:23.526851 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:23.526933 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:23.527572 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Sending DeleteTablet for 3 replicas of tablet 1bcd38163a6f4108ae4ed27893c19115
I20260924 05:04:23.527880 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcd38163a6f4108ae4ed27893c19115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260924 05:04:23.527901 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcd38163a6f4108ae4ed27893c19115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260924 05:04:23.528180 8433 tablet_replica.cc:330] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: stopping tablet replica
I20260924 05:04:23.528187 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcd38163a6f4108ae4ed27893c19115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260924 05:04:23.528270 8433 raft_consensus.cc:2229] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:23.528354 8433 raft_consensus.cc:2258] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:23.528429 8435 tablet_replica.cc:330] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: stopping tablet replica
I20260924 05:04:23.528448 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:23.528591 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:23.528704 8435 raft_consensus.cc:2229] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:23.528784 8433 ts_tablet_manager.cc:1905] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:23.529165 8435 raft_consensus.cc:2258] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:23.529790 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4d336d118f421e90ac4dc3590c7fd8 (DEFAULT_TABLE table=test-table [id=1a752934af3e4ba4aa217a2d451a640a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.529876 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4d336d118f421e90ac4dc3590c7fd8 (DEFAULT_TABLE table=test-table [id=1a752934af3e4ba4aa217a2d451a640a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.529902 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4d336d118f421e90ac4dc3590c7fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.529933 8435 ts_tablet_manager.cc:1905] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:23.529960 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4d336d118f421e90ac4dc3590c7fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.530423 8433 ts_tablet_manager.cc:1918] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:23.530481 8433 log.cc:1196] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/1bcd38163a6f4108ae4ed27893c19115
I20260924 05:04:23.530716 8433 ts_tablet_manager.cc:1939] T 1bcd38163a6f4108ae4ed27893c19115 P 3b753695418240298a35204a7db5f55c: Deleting consensus metadata
I20260924 05:04:23.531075 8411 tablet_bootstrap.cc:492] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Bootstrap starting.
I20260924 05:04:23.531209 7887 catalog_manager.cc:4841] TS 3b753695418240298a35204a7db5f55c (127.0.98.2:33793): tablet 1bcd38163a6f4108ae4ed27893c19115 (table test-table [id=518e80d46f5d45e79d49c0c8d94d5b89]) successfully deleted
I20260924 05:04:23.531397 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4d336d118f421e90ac4dc3590c7fd8 (DEFAULT_TABLE table=test-table [id=1a752934af3e4ba4aa217a2d451a640a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.531462 8411 tablet_bootstrap.cc:654] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.531495 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4d336d118f421e90ac4dc3590c7fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.531765 8414 tablet_bootstrap.cc:492] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Bootstrap starting.
I20260924 05:04:23.532147 8414 tablet_bootstrap.cc:654] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.532545 8411 tablet_bootstrap.cc:492] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: No bootstrap required, opened a new log
I20260924 05:04:23.532593 8411 ts_tablet_manager.cc:1397] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.532617 8410 tablet_bootstrap.cc:492] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Bootstrap starting.
I20260924 05:04:23.532727 8411 raft_consensus.cc:348] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.532793 8411 raft_consensus.cc:374] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.532819 8411 raft_consensus.cc:729] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b753695418240298a35204a7db5f55c, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.532857 8411 consensus_queue.cc:260] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.532943 8411 ts_tablet_manager.cc:1428] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.533228 8414 tablet_bootstrap.cc:492] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: No bootstrap required, opened a new log
I20260924 05:04:23.533248 8410 tablet_bootstrap.cc:654] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.533277 8414 ts_tablet_manager.cc:1397] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.533411 8414 raft_consensus.cc:348] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.533485 8414 raft_consensus.cc:374] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.533506 8414 raft_consensus.cc:729] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2f6b4069ac498f92f3ebe6e5eaed11, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.533550 8414 consensus_queue.cc:260] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.533632 8414 ts_tablet_manager.cc:1428] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:23.534269 8410 tablet_bootstrap.cc:492] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: No bootstrap required, opened a new log
I20260924 05:04:23.534310 8410 ts_tablet_manager.cc:1397] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.534430 8410 raft_consensus.cc:348] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.534480 8410 raft_consensus.cc:374] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.534492 8410 raft_consensus.cc:729] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.534524 8410 consensus_queue.cc:260] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.534598 8410 ts_tablet_manager.cc:1428] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.535043 8435 ts_tablet_manager.cc:1918] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:23.535102 8435 log.cc:1196] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/1bcd38163a6f4108ae4ed27893c19115
I20260924 05:04:23.535307 8435 ts_tablet_manager.cc:1939] T 1bcd38163a6f4108ae4ed27893c19115 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting consensus metadata
I20260924 05:04:23.535808 7886 catalog_manager.cc:4841] TS cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615): tablet 1bcd38163a6f4108ae4ed27893c19115 (table test-table [id=518e80d46f5d45e79d49c0c8d94d5b89]) successfully deleted
I20260924 05:04:23.537813 8434 tablet_replica.cc:330] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: stopping tablet replica
I20260924 05:04:23.537928 8434 raft_consensus.cc:2229] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:23.538066 8434 raft_consensus.cc:2258] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:23.538493 8434 ts_tablet_manager.cc:1905] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:23.539660 8434 ts_tablet_manager.cc:1918] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:23.539726 8434 log.cc:1196] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/1bcd38163a6f4108ae4ed27893c19115
I20260924 05:04:23.539923 8434 ts_tablet_manager.cc:1939] T 1bcd38163a6f4108ae4ed27893c19115 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting consensus metadata
I20260924 05:04:23.540266 7888 catalog_manager.cc:4841] TS 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955): tablet 1bcd38163a6f4108ae4ed27893c19115 (table test-table [id=518e80d46f5d45e79d49c0c8d94d5b89]) successfully deleted
W20260924 05:04:23.546993 8106 tablet.cc:2369] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:04:23.584806 8386 tablet.cc:2369] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:23.634984 8413 raft_consensus.cc:482] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:23.635098 8413 raft_consensus.cc:504] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.635452 8413 leader_election.cc:288] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.638459 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.635611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40615 (local address 127.0.98.1:49205)
0924 05:04:23.635743 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.635754 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.635837 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.636011 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.636012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.636013 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.636052 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.636054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.636817 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.636817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.637759 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.637761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.638255 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.638255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.638316 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.638321 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.638366 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.638401 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260924 05:04:23.638480 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.635708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49205 (local address 127.0.98.3:40615)
0924 05:04:23.635752 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.635754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.635814 (+ 60us) server_negotiation.cc:407] Connection header received
0924 05:04:23.635915 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.635916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.635928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.635972 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.636131 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.636752 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.637826 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.638178 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.638245 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.638256 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.638380 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.638414 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:04:23.638819 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4d336d118f421e90ac4dc3590c7fd8" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" is_pre_election: true
I20260924 05:04:23.638917 8339 raft_consensus.cc:2454] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:23.639130 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.635800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54329 (local address 127.0.98.2:33793)
0924 05:04:23.635838 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.635840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.635876 (+ 36us) server_negotiation.cc:407] Connection header received
0924 05:04:23.635902 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.635903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.635912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.635965 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:23.636851 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.637369 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.638319 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.638814 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.638874 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.638881 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.639026 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.639060 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:23.639086 7990 leader_election.cc:302] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f6f3116f4a4908a45ad65f72a89bd1, cb2f6b4069ac498f92f3ebe6e5eaed11; no voters:
I20260924 05:04:23.639160 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.635611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33793 (local address 127.0.98.1:54329)
0924 05:04:23.635767 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.635776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.635821 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.635974 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.635976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.635977 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:23.636052 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.636055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.637448 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.637449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.638260 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.638262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.638889 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.638890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.638956 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.638960 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.639009 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.639047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260924 05:04:23.639246 8413 raft_consensus.cc:2781] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:23.639277 8413 raft_consensus.cc:482] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:23.639294 8413 raft_consensus.cc:3037] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.639366 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4d336d118f421e90ac4dc3590c7fd8" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c" is_pre_election: true
I20260924 05:04:23.639477 8199 raft_consensus.cc:2454] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:23.639990 8413 raft_consensus.cc:504] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.640097 8413 leader_election.cc:288] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Requested vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.640220 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4d336d118f421e90ac4dc3590c7fd8" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c"
I20260924 05:04:23.640251 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4d336d118f421e90ac4dc3590c7fd8" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
I20260924 05:04:23.640290 8199 raft_consensus.cc:3037] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.640309 8339 raft_consensus.cc:3037] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.640985 8199 raft_consensus.cc:2454] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:23.640985 8339 raft_consensus.cc:2454] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:23.641130 7990 leader_election.cc:302] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f6f3116f4a4908a45ad65f72a89bd1, cb2f6b4069ac498f92f3ebe6e5eaed11; no voters:
I20260924 05:04:23.641242 8413 raft_consensus.cc:2781] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:23.641335 8413 raft_consensus.cc:686] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Running, Role: LEADER
I20260924 05:04:23.641391 8413 consensus_queue.cc:237] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.641986 7902 catalog_manager.cc:5495] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } health_report { overall_health: UNKNOWN } } }
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:23.679757 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.674209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35511 (local address 127.0.0.1:41500)
0924 05:04:23.674338 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.674347 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.674399 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.674498 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.674498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.674772 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:23.674813 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.674815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.675366 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.675366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.675869 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.675870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.676118 (+ 248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.676118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.676152 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.676573 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:23.676577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:23.677817 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:23.678977 (+ 1160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.678982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.678984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.679164 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.679298 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:23.679298 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:23.679298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:23.679344 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:23.679502 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:23.679506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:23.679539 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.679579 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.679659 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260924 05:04:23.679791 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.674268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.0.98.62:35511)
0924 05:04:23.674325 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.674327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.674394 (+ 67us) server_negotiation.cc:407] Connection header received
0924 05:04:23.674438 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.674439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.674446 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.674489 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.674864 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.675319 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.675916 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.676076 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.676119 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.677904 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.677906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.677906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.677913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.678918 (+ 1005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.679215 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.679216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.679216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.679257 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.679380 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.679381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.679381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.679464 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.679506 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.679600 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.679697 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":36}
I20260924 05:04:23.681326 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41500:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 05:04:23.681501 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:23.682705 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 3e76574946fe42c6b2dab8618a54cae3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf158aa0fae24c559fb704dda4308a92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:04:23.682701 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 3e76574946fe42c6b2dab8618a54cae3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf158aa0fae24c559fb704dda4308a92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:04:23.682729 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 3e76574946fe42c6b2dab8618a54cae3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf158aa0fae24c559fb704dda4308a92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:04:23.682832 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e76574946fe42c6b2dab8618a54cae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.682832 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e76574946fe42c6b2dab8618a54cae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.682834 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e76574946fe42c6b2dab8618a54cae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.683825 8410 tablet_bootstrap.cc:492] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: Bootstrap starting.
I20260924 05:04:23.684147 8414 tablet_bootstrap.cc:492] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: Bootstrap starting.
I20260924 05:04:23.684159 8411 tablet_bootstrap.cc:492] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: Bootstrap starting.
I20260924 05:04:23.684191 8410 tablet_bootstrap.cc:654] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.684412 8411 tablet_bootstrap.cc:654] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.684531 8414 tablet_bootstrap.cc:654] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.684885 8410 tablet_bootstrap.cc:492] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: No bootstrap required, opened a new log
I20260924 05:04:23.684902 8411 tablet_bootstrap.cc:492] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: No bootstrap required, opened a new log
I20260924 05:04:23.684938 8410 ts_tablet_manager.cc:1397] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:23.684943 8411 ts_tablet_manager.cc:1397] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:04:23.685063 8411 raft_consensus.cc:348] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.685107 8414 tablet_bootstrap.cc:492] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: No bootstrap required, opened a new log
I20260924 05:04:23.685077 8410 raft_consensus.cc:348] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.685132 8411 raft_consensus.cc:374] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.685139 8410 raft_consensus.cc:374] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.685140 8414 ts_tablet_manager.cc:1397] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:23.685153 8411 raft_consensus.cc:729] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b753695418240298a35204a7db5f55c, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.685159 8410 raft_consensus.cc:729] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.685194 8411 consensus_queue.cc:260] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.685199 8410 consensus_queue.cc:260] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.685280 8410 ts_tablet_manager.cc:1428] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:23.685280 8411 ts_tablet_manager.cc:1428] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.685286 8414 raft_consensus.cc:348] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.685338 8414 raft_consensus.cc:374] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.685354 8414 raft_consensus.cc:729] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2f6b4069ac498f92f3ebe6e5eaed11, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.685387 8414 consensus_queue.cc:260] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.685395 8413 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.685459 8413 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.685459 8414 ts_tablet_manager.cc:1428] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.685604 8420 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.685683 8420 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.685837 8417 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.685920 8417 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.699186 8413 raft_consensus.cc:482] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:23.699244 8413 raft_consensus.cc:504] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.699366 8413 leader_election.cc:288] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.699512 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e76574946fe42c6b2dab8618a54cae3" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c" is_pre_election: true
I20260924 05:04:23.699519 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e76574946fe42c6b2dab8618a54cae3" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" is_pre_election: true
I20260924 05:04:23.699582 8339 raft_consensus.cc:2454] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:23.699584 8199 raft_consensus.cc:2454] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:23.699687 7990 leader_election.cc:302] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f6f3116f4a4908a45ad65f72a89bd1, cb2f6b4069ac498f92f3ebe6e5eaed11; no voters:
I20260924 05:04:23.699780 8413 raft_consensus.cc:2781] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:23.699827 8413 raft_consensus.cc:482] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:23.699846 8413 raft_consensus.cc:3037] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.700410 8413 raft_consensus.cc:504] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.700528 8413 leader_election.cc:288] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Requested vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.700654 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e76574946fe42c6b2dab8618a54cae3" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
I20260924 05:04:23.700719 8339 raft_consensus.cc:3037] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.700700 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e76574946fe42c6b2dab8618a54cae3" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c"
I20260924 05:04:23.700770 8199 raft_consensus.cc:3037] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.701267 8339 raft_consensus.cc:2454] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:23.701288 8199 raft_consensus.cc:2454] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:23.701407 7990 leader_election.cc:302] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f6f3116f4a4908a45ad65f72a89bd1, cb2f6b4069ac498f92f3ebe6e5eaed11; no voters:
I20260924 05:04:23.701483 8413 raft_consensus.cc:2781] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:23.701532 8413 raft_consensus.cc:686] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Running, Role: LEADER
I20260924 05:04:23.701566 8413 consensus_queue.cc:237] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.701835 8441 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: TxnStatusTablet state changed. Reason: New leader 43f6f3116f4a4908a45ad65f72a89bd1. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.701903 8441 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:04:23.702097 8453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 05:04:23.702072 7902 catalog_manager.cc:5495] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:23.702183 8453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 05:04:23.709956 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.707523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.0.98.1:42955)
0924 05:04:23.707563 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.707565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.707576 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:23.707651 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.707652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.707680 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.707731 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:23.707836 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.708307 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.709046 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.709328 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.709384 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.709607 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:23.709745 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:23.709808 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.709879 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.709897 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:23.710036 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.707471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42955 (local address 127.0.0.1:56500)
0924 05:04:23.707537 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.707548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.707592 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.707733 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.707734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.707735 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:23.707778 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.707780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.708373 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.708374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.708949 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.708952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.709382 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.709383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.709470 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.709499 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:23.709794 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:23.709803 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.709874 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.709946 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:04:23.713754 8339 raft_consensus.cc:1261] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:23.713840 8199 raft_consensus.cc:1261] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:23.713965 8413 consensus_queue.cc:1035] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:23.714061 8441 consensus_queue.cc:1035] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:23.714605 8420 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: TxnStatusTablet state changed. Reason: New leader 43f6f3116f4a4908a45ad65f72a89bd1. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.714681 8420 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.715232 8417 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: TxnStatusTablet state changed. Reason: New leader 43f6f3116f4a4908a45ad65f72a89bd1. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.715306 8417 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.715813 8413 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.715883 8413 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:04:23.716017 8417 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.716081 8417 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P cb2f6b4069ac498f92f3ebe6e5eaed11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.717399 8441 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.717474 8441 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 43f6f3116f4a4908a45ad65f72a89bd1: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:04:23.718163 8420 tablet_replica.cc:439] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } } }
I20260924 05:04:23.718240 8420 tablet_replica.cc:442] T 3e76574946fe42c6b2dab8618a54cae3 P 3b753695418240298a35204a7db5f55c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:04:23.720532 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.717843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.98.1:42955)
0924 05:04:23.717885 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.717887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.718647 (+ 760us) server_negotiation.cc:407] Connection header received
0924 05:04:23.718666 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.718667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.718676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.718720 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:23.718836 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.719267 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.719951 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.720139 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.720189 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.720315 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:23.720345 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:23.720383 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.720450 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.720464 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:23.724898 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.722792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44765 (local address 127.0.98.1:42955)
0924 05:04:23.722823 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.722825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.722942 (+ 117us) server_negotiation.cc:407] Connection header received
0924 05:04:23.722986 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.722986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.722994 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.723035 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:23.723146 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.723595 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.724317 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.724480 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.724536 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.724668 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:23.724697 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:23.724735 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.724785 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.724814 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":14}
I20260924 05:04:23.724974 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.722749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42955 (local address 127.0.98.1:44765)
0924 05:04:23.722883 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:23.722892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:23.722945 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:23.723038 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:23.723038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:23.723049 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:04:23.723096 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.723097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.723657 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.723658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.724246 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:23.724247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.724532 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.724533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:23.724589 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.724602 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:04:23.724742 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:04:23.724744 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:23.724784 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:04:23.724853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":14,"thread_start_us":83,"threads_started":1}
I20260924 05:04:23.726909 8339 raft_consensus.cc:1261] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:23.726940 8199 raft_consensus.cc:1261] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:23.727123 8461 consensus_queue.cc:1035] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:23.727224 8441 consensus_queue.cc:1035] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:23.740688 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:23.740792 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:23.741328 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Sending DeleteTablet for 3 replicas of tablet 1b4d336d118f421e90ac4dc3590c7fd8
I20260924 05:04:23.741559 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4d336d118f421e90ac4dc3590c7fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260924 05:04:23.741662 8433 tablet_replica.cc:330] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: stopping tablet replica
I20260924 05:04:23.741753 8433 raft_consensus.cc:2229] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:23.741791 8433 raft_consensus.cc:2258] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:23.741781 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4d336d118f421e90ac4dc3590c7fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260924 05:04:23.741861 8434 tablet_replica.cc:330] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: stopping tablet replica
I20260924 05:04:23.741909 8434 raft_consensus.cc:2229] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:23.741982 8434 raft_consensus.cc:2258] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:23.742048 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4d336d118f421e90ac4dc3590c7fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260924 05:04:23.742148 8435 tablet_replica.cc:330] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: stopping tablet replica
I20260924 05:04:23.742211 8435 raft_consensus.cc:2229] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:23.742214 8434 ts_tablet_manager.cc:1905] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:23.742250 8435 raft_consensus.cc:2258] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:23.742590 8433 ts_tablet_manager.cc:1905] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:23.743707 8434 ts_tablet_manager.cc:1918] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:23.743753 8434 log.cc:1196] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/1b4d336d118f421e90ac4dc3590c7fd8
I20260924 05:04:23.743937 8434 ts_tablet_manager.cc:1939] T 1b4d336d118f421e90ac4dc3590c7fd8 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting consensus metadata
I20260924 05:04:23.744056 8433 ts_tablet_manager.cc:1918] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:23.744096 8433 log.cc:1196] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/1b4d336d118f421e90ac4dc3590c7fd8
I20260924 05:04:23.744266 8433 ts_tablet_manager.cc:1939] T 1b4d336d118f421e90ac4dc3590c7fd8 P 3b753695418240298a35204a7db5f55c: Deleting consensus metadata
I20260924 05:04:23.744510 8435 ts_tablet_manager.cc:1905] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:23.744616 7887 catalog_manager.cc:4841] TS 3b753695418240298a35204a7db5f55c (127.0.98.2:33793): tablet 1b4d336d118f421e90ac4dc3590c7fd8 (table test-table [id=1a752934af3e4ba4aa217a2d451a640a]) successfully deleted
I20260924 05:04:23.745096 7888 catalog_manager.cc:4841] TS 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955): tablet 1b4d336d118f421e90ac4dc3590c7fd8 (table test-table [id=1a752934af3e4ba4aa217a2d451a640a]) successfully deleted
I20260924 05:04:23.745576 8435 ts_tablet_manager.cc:1918] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:23.745609 8435 log.cc:1196] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/1b4d336d118f421e90ac4dc3590c7fd8
I20260924 05:04:23.745764 8435 ts_tablet_manager.cc:1939] T 1b4d336d118f421e90ac4dc3590c7fd8 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting consensus metadata
I20260924 05:04:23.746058 7886 catalog_manager.cc:4841] TS cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615): tablet 1b4d336d118f421e90ac4dc3590c7fd8 (table test-table [id=1a752934af3e4ba4aa217a2d451a640a]) successfully deleted
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:04:23.772543 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:23.767539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.0.98.62:35511)
0924 05:04:23.767579 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.767581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.767711 (+ 130us) server_negotiation.cc:407] Connection header received
0924 05:04:23.767793 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.767794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.767803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.767847 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.768372 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.768900 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.769601 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.769847 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.769897 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.772485 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55004: BlockingRecv error: recv got EOF from 127.0.0.1:55004 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 05:04:23 dist-test-slave-m01t krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:04:23.779731 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:23.773303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.0.98.62:35511)
0924 05:04:23.773342 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:23.773344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:23.773376 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:04:23.773423 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:23.773424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:23.773433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:23.773500 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:23.773935 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.774632 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.775247 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:23.775518 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:23.775584 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:23.777609 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:23.777610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:23.777610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:23.777616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:23.778811 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.779164 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.779165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.779165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.779206 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:23.779364 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:23.779364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:23.779364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:23.779447 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:23.779509 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:23.779627 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:04:23.779675 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:04:23.781056 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:23.781225 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:23.782400 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 44bf413a38394584b6a0ac94a649ddc0 (DEFAULT_TABLE table=test-table [id=cabd259f9c2a4d2fa698387a0cac0b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.782411 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 44bf413a38394584b6a0ac94a649ddc0 (DEFAULT_TABLE table=test-table [id=cabd259f9c2a4d2fa698387a0cac0b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.782442 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 44bf413a38394584b6a0ac94a649ddc0 (DEFAULT_TABLE table=test-table [id=cabd259f9c2a4d2fa698387a0cac0b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:23.782523 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf413a38394584b6a0ac94a649ddc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.782524 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf413a38394584b6a0ac94a649ddc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.782542 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bf413a38394584b6a0ac94a649ddc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:23.783689 8410 tablet_bootstrap.cc:492] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Bootstrap starting.
I20260924 05:04:23.783689 8414 tablet_bootstrap.cc:492] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Bootstrap starting.
I20260924 05:04:23.783831 8411 tablet_bootstrap.cc:492] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Bootstrap starting.
I20260924 05:04:23.784107 8410 tablet_bootstrap.cc:654] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.784132 8411 tablet_bootstrap.cc:654] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.784252 8414 tablet_bootstrap.cc:654] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:23.784929 8411 tablet_bootstrap.cc:492] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: No bootstrap required, opened a new log
I20260924 05:04:23.784986 8411 ts_tablet_manager.cc:1397] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:23.785005 8410 tablet_bootstrap.cc:492] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: No bootstrap required, opened a new log
I20260924 05:04:23.785046 8410 ts_tablet_manager.cc:1397] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:23.785132 8411 raft_consensus.cc:348] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.785202 8411 raft_consensus.cc:374] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.785228 8411 raft_consensus.cc:729] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b753695418240298a35204a7db5f55c, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.785180 8410 raft_consensus.cc:348] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.785252 8410 raft_consensus.cc:374] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.785276 8410 raft_consensus.cc:729] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.785266 8411 consensus_queue.cc:260] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.785313 8410 consensus_queue.cc:260] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.785348 8411 ts_tablet_manager.cc:1428] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.785385 8410 ts_tablet_manager.cc:1428] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:23.785740 8414 tablet_bootstrap.cc:492] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: No bootstrap required, opened a new log
I20260924 05:04:23.785785 8414 ts_tablet_manager.cc:1397] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:23.785914 8414 raft_consensus.cc:348] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.785977 8414 raft_consensus.cc:374] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:23.785996 8414 raft_consensus.cc:729] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2f6b4069ac498f92f3ebe6e5eaed11, State: Initialized, Role: FOLLOWER
I20260924 05:04:23.786057 8414 consensus_queue.cc:260] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.786164 8414 ts_tablet_manager.cc:1428] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:23.971035 8461 raft_consensus.cc:482] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:23.971119 8461 raft_consensus.cc:504] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.971269 8461 leader_election.cc:288] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.971441 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf413a38394584b6a0ac94a649ddc0" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c" is_pre_election: true
I20260924 05:04:23.971458 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf413a38394584b6a0ac94a649ddc0" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" is_pre_election: true
I20260924 05:04:23.971530 8199 raft_consensus.cc:2454] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:23.971549 8339 raft_consensus.cc:2454] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:23.971668 7990 leader_election.cc:302] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f6f3116f4a4908a45ad65f72a89bd1, cb2f6b4069ac498f92f3ebe6e5eaed11; no voters:
I20260924 05:04:23.971774 8461 raft_consensus.cc:2781] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:23.971817 8461 raft_consensus.cc:482] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:23.971830 8461 raft_consensus.cc:3037] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.972522 8461 raft_consensus.cc:504] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.972631 8461 leader_election.cc:288] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Requested vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:23.972760 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf413a38394584b6a0ac94a649ddc0" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
I20260924 05:04:23.972781 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bf413a38394584b6a0ac94a649ddc0" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c"
I20260924 05:04:23.972826 8339 raft_consensus.cc:3037] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.972841 8199 raft_consensus.cc:3037] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:23.973449 8199 raft_consensus.cc:2454] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:23.973457 8339 raft_consensus.cc:2454] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:23.973591 7991 leader_election.cc:302] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b753695418240298a35204a7db5f55c, 43f6f3116f4a4908a45ad65f72a89bd1; no voters:
I20260924 05:04:23.973687 8461 raft_consensus.cc:2781] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:23.973748 8461 raft_consensus.cc:686] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Running, Role: LEADER
I20260924 05:04:23.973789 8461 consensus_queue.cc:237] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:23.974252 7893 catalog_manager.cc:5495] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:24.011014 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:24.009177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.0.98.1:42955)
0924 05:04:24.009221 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:24.009223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:24.009231 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:24.009268 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:24.009269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:24.009278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:24.009320 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:24.009406 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:24.009896 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:24.010560 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:24.010728 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:24.010768 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:24.010857 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:24.010881 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:24.010911 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:24.010945 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:04:24.010958 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:04:24.011713 8199 raft_consensus.cc:1261] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:24.011726 8339 raft_consensus.cc:1261] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:24.011924 8441 consensus_queue.cc:1035] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:24.012033 8461 consensus_queue.cc:1035] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:24.014261 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:24.014333 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:24.014801 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Sending DeleteTablet for 3 replicas of tablet 44bf413a38394584b6a0ac94a649ddc0
I20260924 05:04:24.015012 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bf413a38394584b6a0ac94a649ddc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260924 05:04:24.015012 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bf413a38394584b6a0ac94a649ddc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260924 05:04:24.015008 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bf413a38394584b6a0ac94a649ddc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260924 05:04:24.015093 8433 tablet_replica.cc:330] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: stopping tablet replica
I20260924 05:04:24.015110 8434 tablet_replica.cc:330] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: stopping tablet replica
I20260924 05:04:24.015144 8433 raft_consensus.cc:2229] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:24.015169 8434 raft_consensus.cc:2229] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:24.015184 8433 raft_consensus.cc:2258] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:24.015228 8434 raft_consensus.cc:2258] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:24.015297 8435 tablet_replica.cc:330] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: stopping tablet replica
I20260924 05:04:24.015348 8435 raft_consensus.cc:2229] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:24.015380 8435 raft_consensus.cc:2258] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:24.015453 8434 ts_tablet_manager.cc:1905] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:24.015635 8435 ts_tablet_manager.cc:1905] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:24.015674 8433 ts_tablet_manager.cc:1905] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:24.015632 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:24.015753 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:24.016820 8434 ts_tablet_manager.cc:1918] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:24.016866 8434 log.cc:1196] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/44bf413a38394584b6a0ac94a649ddc0
I20260924 05:04:24.016908 8435 ts_tablet_manager.cc:1918] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:24.016906 8433 ts_tablet_manager.cc:1918] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:24.016947 8435 log.cc:1196] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/44bf413a38394584b6a0ac94a649ddc0
I20260924 05:04:24.016955 8433 log.cc:1196] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/44bf413a38394584b6a0ac94a649ddc0
I20260924 05:04:24.017055 8434 ts_tablet_manager.cc:1939] T 44bf413a38394584b6a0ac94a649ddc0 P 43f6f3116f4a4908a45ad65f72a89bd1: Deleting consensus metadata
I20260924 05:04:24.017127 8433 ts_tablet_manager.cc:1939] T 44bf413a38394584b6a0ac94a649ddc0 P 3b753695418240298a35204a7db5f55c: Deleting consensus metadata
I20260924 05:04:24.017123 8435 ts_tablet_manager.cc:1939] T 44bf413a38394584b6a0ac94a649ddc0 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting consensus metadata
I20260924 05:04:24.017506 7887 catalog_manager.cc:4841] TS 3b753695418240298a35204a7db5f55c (127.0.98.2:33793): tablet 44bf413a38394584b6a0ac94a649ddc0 (table test-table [id=cabd259f9c2a4d2fa698387a0cac0b8f]) successfully deleted
I20260924 05:04:24.017576 7888 catalog_manager.cc:4841] TS 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955): tablet 44bf413a38394584b6a0ac94a649ddc0 (table test-table [id=cabd259f9c2a4d2fa698387a0cac0b8f]) successfully deleted
I20260924 05:04:24.017761 7886 catalog_manager.cc:4841] TS cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615): tablet 44bf413a38394584b6a0ac94a649ddc0 (table test-table [id=cabd259f9c2a4d2fa698387a0cac0b8f]) successfully deleted
I20260924 05:04:24.017904 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 48f653de25794299b1bab2c5c2f89e98 (DEFAULT_TABLE table=test-table [id=57aaaada24f443c192f94fd19f0f1f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:24.017915 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 48f653de25794299b1bab2c5c2f89e98 (DEFAULT_TABLE table=test-table [id=57aaaada24f443c192f94fd19f0f1f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:24.017953 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 48f653de25794299b1bab2c5c2f89e98 (DEFAULT_TABLE table=test-table [id=57aaaada24f443c192f94fd19f0f1f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:24.017989 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f653de25794299b1bab2c5c2f89e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:24.018006 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f653de25794299b1bab2c5c2f89e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:24.018033 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f653de25794299b1bab2c5c2f89e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:24.018715 8410 tablet_bootstrap.cc:492] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1: Bootstrap starting.
I20260924 05:04:24.018991 8410 tablet_bootstrap.cc:654] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:24.019050 8414 tablet_bootstrap.cc:492] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Bootstrap starting.
I20260924 05:04:24.019062 8411 tablet_bootstrap.cc:492] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Bootstrap starting.
I20260924 05:04:24.019364 8411 tablet_bootstrap.cc:654] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:24.019367 8414 tablet_bootstrap.cc:654] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:24.019795 8411 tablet_bootstrap.cc:492] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: No bootstrap required, opened a new log
I20260924 05:04:24.019804 8410 tablet_bootstrap.cc:492] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1: No bootstrap required, opened a new log
I20260924 05:04:24.019847 8410 ts_tablet_manager.cc:1397] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:04:24.019847 8411 ts_tablet_manager.cc:1397] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:04:24.019945 8410 raft_consensus.cc:348] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.019946 8411 raft_consensus.cc:348] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.020008 8410 raft_consensus.cc:374] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:24.020026 8410 raft_consensus.cc:729] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Initialized, Role: FOLLOWER
I20260924 05:04:24.020035 8411 raft_consensus.cc:374] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:24.020068 8411 raft_consensus.cc:729] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b753695418240298a35204a7db5f55c, State: Initialized, Role: FOLLOWER
I20260924 05:04:24.020062 8410 consensus_queue.cc:260] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.020112 8411 consensus_queue.cc:260] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.020148 8410 ts_tablet_manager.cc:1428] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:24.020200 8411 ts_tablet_manager.cc:1428] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:24.020509 8414 tablet_bootstrap.cc:492] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: No bootstrap required, opened a new log
I20260924 05:04:24.020568 8414 ts_tablet_manager.cc:1397] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:24.020701 8414 raft_consensus.cc:348] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.020767 8414 raft_consensus.cc:374] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:24.020788 8414 raft_consensus.cc:729] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2f6b4069ac498f92f3ebe6e5eaed11, State: Initialized, Role: FOLLOWER
I20260924 05:04:24.020829 8414 consensus_queue.cc:260] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.020911 8414 ts_tablet_manager.cc:1428] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:24.022456 8441 raft_consensus.cc:482] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:24.022503 8441 raft_consensus.cc:504] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.022629 8441 leader_election.cc:288] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:24.022754 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f653de25794299b1bab2c5c2f89e98" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" is_pre_election: true
I20260924 05:04:24.022738 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f653de25794299b1bab2c5c2f89e98" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c" is_pre_election: true
I20260924 05:04:24.022819 8339 raft_consensus.cc:2454] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:24.022819 8199 raft_consensus.cc:2454] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 0.
I20260924 05:04:24.022943 7991 leader_election.cc:302] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b753695418240298a35204a7db5f55c, 43f6f3116f4a4908a45ad65f72a89bd1; no voters:
I20260924 05:04:24.023029 8441 raft_consensus.cc:2781] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:24.023072 8441 raft_consensus.cc:482] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:24.023084 8441 raft_consensus.cc:3037] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:24.023622 8441 raft_consensus.cc:504] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.023722 8441 leader_election.cc:288] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Requested vote from peers 3b753695418240298a35204a7db5f55c (127.0.98.2:33793), cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615)
I20260924 05:04:24.023833 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f653de25794299b1bab2c5c2f89e98" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11"
I20260924 05:04:24.023897 8339 raft_consensus.cc:3037] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:24.024426 8339 raft_consensus.cc:2454] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:24.024505 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f653de25794299b1bab2c5c2f89e98" candidate_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b753695418240298a35204a7db5f55c"
I20260924 05:04:24.024564 8199 raft_consensus.cc:3037] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:24.024575 7990 leader_election.cc:302] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f6f3116f4a4908a45ad65f72a89bd1, cb2f6b4069ac498f92f3ebe6e5eaed11; no voters:
I20260924 05:04:24.024657 8441 raft_consensus.cc:2781] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:24.024705 8441 raft_consensus.cc:686] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Becoming Leader. State: Replica: 43f6f3116f4a4908a45ad65f72a89bd1, State: Running, Role: LEADER
I20260924 05:04:24.024752 8441 consensus_queue.cc:237] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } }
I20260924 05:04:24.025223 8199 raft_consensus.cc:2454] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f6f3116f4a4908a45ad65f72a89bd1 in term 1.
I20260924 05:04:24.025177 7902 catalog_manager.cc:5495] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43f6f3116f4a4908a45ad65f72a89bd1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:24.030913 8339 raft_consensus.cc:1261] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:24.031033 8199 raft_consensus.cc:1261] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Refusing update from remote peer 43f6f3116f4a4908a45ad65f72a89bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:24.031121 8441 consensus_queue.cc:1035] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb2f6b4069ac498f92f3ebe6e5eaed11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:24.031208 8413 consensus_queue.cc:1035] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b753695418240298a35204a7db5f55c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:24.039863 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:24.039959 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:24.040479 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2896f40bb0940c2be9fbff58083d9b0: Sending DeleteTablet for 3 replicas of tablet 48f653de25794299b1bab2c5c2f89e98
I20260924 05:04:24.040706 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f653de25794299b1bab2c5c2f89e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57800
I20260924 05:04:24.040812 8433 tablet_replica.cc:330] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: stopping tablet replica
I20260924 05:04:24.040802 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f653de25794299b1bab2c5c2f89e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52216
I20260924 05:04:24.040874 8433 raft_consensus.cc:2229] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:24.040894 8435 tablet_replica.cc:330] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: stopping tablet replica
I20260924 05:04:24.040915 8433 raft_consensus.cc:2258] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:24.040930 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f653de25794299b1bab2c5c2f89e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260924 05:04:24.040957 8435 raft_consensus.cc:2229] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:24.040992 8435 raft_consensus.cc:2258] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:24.041011 8434 tablet_replica.cc:330] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1: stopping tablet replica
I20260924 05:04:24.041059 8434 raft_consensus.cc:2229] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:24.041122 8434 raft_consensus.cc:2258] T 48f653de25794299b1bab2c5c2f89e98 P 43f6f3116f4a4908a45ad65f72a89bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:24.041252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 7974
I20260924 05:04:24.041297 8435 ts_tablet_manager.cc:1905] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:24.041316 8433 ts_tablet_manager.cc:1905] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:24.042555 8435 ts_tablet_manager.cc:1918] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:24.042608 8435 log.cc:1196] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/48f653de25794299b1bab2c5c2f89e98
I20260924 05:04:24.042797 8435 ts_tablet_manager.cc:1939] T 48f653de25794299b1bab2c5c2f89e98 P cb2f6b4069ac498f92f3ebe6e5eaed11: Deleting consensus metadata
I20260924 05:04:24.043046 7886 catalog_manager.cc:4841] TS cb2f6b4069ac498f92f3ebe6e5eaed11 (127.0.98.3:40615): tablet 48f653de25794299b1bab2c5c2f89e98 (table test-table [id=57aaaada24f443c192f94fd19f0f1f6d]) successfully deleted
I20260924 05:04:24.043236 8433 ts_tablet_manager.cc:1918] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:04:24.043280 8433 log.cc:1196] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/48f653de25794299b1bab2c5c2f89e98
I20260924 05:04:24.043453 8433 ts_tablet_manager.cc:1939] T 48f653de25794299b1bab2c5c2f89e98 P 3b753695418240298a35204a7db5f55c: Deleting consensus metadata
I20260924 05:04:24.043794 7887 catalog_manager.cc:4841] TS 3b753695418240298a35204a7db5f55c (127.0.98.2:33793): tablet 48f653de25794299b1bab2c5c2f89e98 (table test-table [id=57aaaada24f443c192f94fd19f0f1f6d]) successfully deleted
W20260924 05:04:24.047705 7888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42955 (error 108)
I20260924 05:04:24.047864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 8114
I20260924 05:04:24.048418 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:24.047995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42955 (local address 127.0.0.1:56510)
0924 05:04:24.048343 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:24.048382 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":34,"threads_started":1}
W20260924 05:04:24.048498 7888 catalog_manager.cc:4584] TS 43f6f3116f4a4908a45ad65f72a89bd1 (127.0.98.1:42955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48f653de25794299b1bab2c5c2f89e98: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42955: connect: Connection refused (error 111)
I20260924 05:04:24.052551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 8254
I20260924 05:04:24.056303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 7870
2026-09-24T05:04:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2372 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:24 dist-test-slave-m01t krb5kdc[8494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:24 dist-test-slave-m01t krb5kdc[8494](info): set up 2 sockets
Sep 24 05:04:24 dist-test-slave-m01t krb5kdc[8494](info): commencing operation
krb5kdc: starting...
W20260924 05:04:24.997890 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:25 dist-test-slave-m01t krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:25.101346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:25.177912 8507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:25.178112 8507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:25.178140 8507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:25.179257 8507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:25.179294 8507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:25.179312 8507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:25.179327 8507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:25.179342 8507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:25.180814 8507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:25.181049 8507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:25.181241 8507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:04:25.185580 8507 server_base.cc:1034] running on GCE node
W20260924 05:04:25.185492 8512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:25.185516 8513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:25.185496 8515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:25.185973 8507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:25.186215 8507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:25.187343 8507 hybrid_clock.cc:648] HybridClock initialized: now 1790226265187331 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T05:04:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1132 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:25 dist-test-slave-m01t krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:25 dist-test-slave-m01t krb5kdc[8520](info): set up 2 sockets
Sep 24 05:04:25 dist-test-slave-m01t krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260924 05:04:26.134096 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:26.236256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:26.313117 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:26.313309 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:26.313330 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:26.314481 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:26.314519 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:26.314531 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:26.314555 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:26.314579 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:26.316025 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:26.316236 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:26.316418 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:26.320626 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.320660 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.320665 8533 server_base.cc:1034] running on GCE node
W20260924 05:04:26.320636 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.320981 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:26.321223 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:26.322353 8533 hybrid_clock.cc:648] HybridClock initialized: now 1790226266322339 us; error 29 us; skew 500 ppm
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:26.324563 8533 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:26.324956 8533 webserver.cc:460] Webserver started at http://127.0.98.62:43113/ using document root <none> and password file <none>
I20260924 05:04:26.325148 8533 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:26.325193 8533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:26.325314 8533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:26.326354 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "2c7f05beb2aa4e0bb757f66836de287e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "cc11f7cd4b42fc38761ec36fb9b151df"
server_key_iv: "758143c78ba107618836510dd2912cfe"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.326684 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c7f05beb2aa4e0bb757f66836de287e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "cc11f7cd4b42fc38761ec36fb9b151df"
server_key_iv: "758143c78ba107618836510dd2912cfe"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.327847 8533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:26.328471 8548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.328619 8533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:26.328696 8533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "2c7f05beb2aa4e0bb757f66836de287e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "cc11f7cd4b42fc38761ec36fb9b151df"
server_key_iv: "758143c78ba107618836510dd2912cfe"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.328754 8533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:26.365036 8533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:26.366765 8533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:26.366895 8533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:26.370536 8533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35395
I20260924 05:04:26.370602 8600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35395 every 8 connection(s)
I20260924 05:04:26.371021 8533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:26.371622 8601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:26.373818 8601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Bootstrap starting.
I20260924 05:04:26.374356 8601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:26.374619 8601 log.cc:824] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:26.375191 8601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: No bootstrap required, opened a new log
I20260924 05:04:26.376686 8601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:26.376816 8601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:26.376847 8601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7f05beb2aa4e0bb757f66836de287e, State: Initialized, Role: FOLLOWER
I20260924 05:04:26.376984 8601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:26.377043 8601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:26.377063 8601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:26.377108 8601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:26.377948 8601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:26.378038 8601 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7f05beb2aa4e0bb757f66836de287e; no voters:
I20260924 05:04:26.378183 8601 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:26.378217 8604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:26.378348 8604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 LEADER]: Becoming Leader. State: Replica: 2c7f05beb2aa4e0bb757f66836de287e, State: Running, Role: LEADER
I20260924 05:04:26.378400 8601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:26.378469 8604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:26.378794 8606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7f05beb2aa4e0bb757f66836de287e. Latest consensus state: current_term: 1 leader_uuid: "2c7f05beb2aa4e0bb757f66836de287e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } } }
I20260924 05:04:26.378793 8605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c7f05beb2aa4e0bb757f66836de287e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } } }
I20260924 05:04:26.378983 8605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:26.378973 8606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:26.379350 8613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:26.379810 8613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:26.380824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 8533
I20260924 05:04:26.380929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:26.381108 392 external_mini_cluster.cc:1428] Setting key e63bdde76168d6125c34e945939b7bf5
I20260924 05:04:26.381206 8613 catalog_manager.cc:1347] Generated new cluster ID: 4d9db7be25014a8abdf8265c910daa3d
I20260924 05:04:26.381238 8613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:26.389360 8613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:26.390010 8613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:26.392174 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.381536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.0.98.62:35395)
0924 05:04:26.381837 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:26.381847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:26.381877 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:04:26.381995 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:26.382001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:26.382053 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:26.382907 (+ 854us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:26.383011 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.384380 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.385440 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.386073 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.386459 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.388782 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:26.388786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:26.388786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:26.388797 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:26.390719 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.391089 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.391092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.391092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.391188 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.391360 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.391361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.391361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.391600 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:26.391738 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:26.391824 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:04:26.391957 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":67,"threads_started":1}
I20260924 05:04:26.392889 8613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Generated new TSK 0
I20260924 05:04:26.393038 8613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:26.420121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35395
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:26.508893 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:26.509073 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:26.509101 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:26.510215 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:26.510253 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:26.510288 8627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:26.511770 8627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:26.511982 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:26.512166 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:26.517406 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.517467 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.517473 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.517730 8627 server_base.cc:1034] running on GCE node
I20260924 05:04:26.517923 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:26.518131 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:26.519258 8627 hybrid_clock.cc:648] HybridClock initialized: now 1790226266519246 us; error 28 us; skew 500 ppm
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:26.521504 8627 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:26.521950 8627 webserver.cc:460] Webserver started at http://127.0.98.1:43555/ using document root <none> and password file <none>
I20260924 05:04:26.522161 8627 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:26.522202 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:26.522318 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:26.523288 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ac999b9e5404ce1bde9e0e425ce716e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "54895196b49b27667044ea38eb9704df"
server_key_iv: "605410937555e991e973e282645220df"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.523597 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ac999b9e5404ce1bde9e0e425ce716e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "54895196b49b27667044ea38eb9704df"
server_key_iv: "605410937555e991e973e282645220df"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.524765 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:26.525405 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.525583 8627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:26.525666 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "3ac999b9e5404ce1bde9e0e425ce716e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "54895196b49b27667044ea38eb9704df"
server_key_iv: "605410937555e991e973e282645220df"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.525774 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:26.535677 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:26.537868 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:26.537999 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:26.538208 8627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:26.538496 8627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:26.538530 8627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.538560 8627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:26.538582 8627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.543850 8627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39751
I20260924 05:04:26.543893 8755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39751 every 8 connection(s)
I20260924 05:04:26.544169 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:26.544432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 8627
I20260924 05:04:26.544556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:26.544771 392 external_mini_cluster.cc:1428] Setting key 7ea37bbc9eb10d4c5a6ec012c1bd2ef5
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:26.553165 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.544851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54003 (local address 127.0.98.62:35395)
0924 05:04:26.544895 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:26.544898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:26.545121 (+ 223us) server_negotiation.cc:407] Connection header received
0924 05:04:26.546115 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:26.546116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:26.546132 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:26.546193 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:26.546855 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.547343 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.548271 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.548449 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.548494 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.550634 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:26.550635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:26.550636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:26.550643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:26.551980 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.552319 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.552320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.552320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.552375 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.552568 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.552569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.552569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.552704 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:26.552772 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:26.552980 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 05:04:26.553068 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:26.553210 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.544846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:54003)
0924 05:04:26.545003 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:26.545048 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:26.546020 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:26.546249 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:26.546252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:26.546490 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:26.546770 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:26.546776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.547426 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.547426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:26.548207 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:26.548209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.548513 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.548514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:26.548649 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.549090 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:26.549100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:26.550546 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:26.552052 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:26.552057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:26.552059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:26.552254 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:26.552430 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:26.552431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:26.552431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:26.552513 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:26.552761 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:26.552763 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:26.552866 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 05:04:26.552938 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:04:26.553057 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:04:26.553690 8756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35395
I20260924 05:04:26.553802 8756 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:26.553961 8756 heartbeater.cc:511] Master 127.0.98.62:35395 requested a full tablet report, sending...
I20260924 05:04:26.554445 8565 ts_manager.cc:194] Registered new tserver with Master: 3ac999b9e5404ce1bde9e0e425ce716e (127.0.98.1:39751)
I20260924 05:04:26.555027 8565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:26.567593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35395
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:26.644842 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:26.645035 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:26.645056 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:26.646157 8761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:26.646190 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:26.646214 8761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:26.647596 8761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:26.647773 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:26.647938 8761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:26.652199 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.652215 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.652205 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.652483 8761 server_base.cc:1034] running on GCE node
I20260924 05:04:26.652699 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:26.652921 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:26.654057 8761 hybrid_clock.cc:648] HybridClock initialized: now 1790226266654042 us; error 25 us; skew 500 ppm
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:26.656332 8761 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:26.656693 8761 webserver.cc:460] Webserver started at http://127.0.98.2:38219/ using document root <none> and password file <none>
I20260924 05:04:26.656855 8761 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:26.656893 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:26.656977 8761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:26.657929 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9331114d1a5424486b9ef2d5da92df4"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "232ad3fc2f37956dc598f1cea3581426"
server_key_iv: "dd92838acc5047fb6f6e001cc72f89e3"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.658217 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9331114d1a5424486b9ef2d5da92df4"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "232ad3fc2f37956dc598f1cea3581426"
server_key_iv: "dd92838acc5047fb6f6e001cc72f89e3"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.659291 8761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:26.659861 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.660004 8761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:26.660077 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "a9331114d1a5424486b9ef2d5da92df4"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "232ad3fc2f37956dc598f1cea3581426"
server_key_iv: "dd92838acc5047fb6f6e001cc72f89e3"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.660132 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:26.669647 8761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:26.671193 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:26.671286 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:26.671437 8761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:26.671684 8761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:26.671713 8761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.671731 8761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:26.671754 8761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.676966 8761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32899
I20260924 05:04:26.677024 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32899 every 8 connection(s)
I20260924 05:04:26.677264 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:26.681133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 8761
I20260924 05:04:26.681241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:26.681437 392 external_mini_cluster.cc:1428] Setting key 0900f9d6051dbf47efb2dbe489723e0c
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:26.685622 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.677895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43979 (local address 127.0.98.62:35395)
0924 05:04:26.677959 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:26.677961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:26.678111 (+ 150us) server_negotiation.cc:407] Connection header received
0924 05:04:26.679004 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:26.679005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:26.679014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:26.679060 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:26.679567 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.680092 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.680996 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.681272 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.681338 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.683217 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:26.683218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:26.683218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:26.683224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:26.684381 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.684802 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.684803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.684803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.684848 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.685091 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.685091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.685091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.685183 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:26.685225 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:26.685509 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 05:04:26.685560 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:04:26.685895 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.677877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:43979)
0924 05:04:26.678001 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:26.678034 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:26.678924 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:26.679112 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:26.679115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:26.679328 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:26.679495 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:26.679498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.680158 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.680159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:26.680932 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:26.680934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.681342 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.681343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:26.681440 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.681854 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:26.681862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:26.683134 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:26.684454 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:26.684460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:26.684462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:26.684737 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:26.684904 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:26.684905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:26.684905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:26.685036 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:26.685235 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:26.685240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:26.685389 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 05:04:26.685500 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 05:04:26.685646 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260924 05:04:26.686321 8890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35395
I20260924 05:04:26.686405 8890 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:26.686575 8890 heartbeater.cc:511] Master 127.0.98.62:35395 requested a full tablet report, sending...
I20260924 05:04:26.686909 8565 ts_manager.cc:194] Registered new tserver with Master: a9331114d1a5424486b9ef2d5da92df4 (127.0.98.2:32899)
I20260924 05:04:26.687258 8565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43979
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:26.702225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35395
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:26.778954 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:26.779170 8895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:26.779196 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:26.780280 8895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:26.780315 8895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:26.780339 8895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:26.781745 8895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:26.781901 8895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:26.782055 8895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:26.786325 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.786325 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.786387 8895 server_base.cc:1034] running on GCE node
W20260924 05:04:26.786326 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.786639 8895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:26.786854 8895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:26.787979 8895 hybrid_clock.cc:648] HybridClock initialized: now 1790226266787966 us; error 26 us; skew 500 ppm
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:26.790141 8895 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:26.790517 8895 webserver.cc:460] Webserver started at http://127.0.98.3:43587/ using document root <none> and password file <none>
I20260924 05:04:26.790696 8895 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:26.790740 8895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:26.790855 8895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:26.791792 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "054a44d5667b47968955afc679594d3c"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "2531675bc5a5e7523c3a63244c86ab01"
server_key_iv: "6d0020c22b544328645aa6152b86d557"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.792111 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "054a44d5667b47968955afc679594d3c"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "2531675bc5a5e7523c3a63244c86ab01"
server_key_iv: "6d0020c22b544328645aa6152b86d557"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.793217 8895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:26.793816 8910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.793944 8895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:26.794021 8895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "054a44d5667b47968955afc679594d3c"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "2531675bc5a5e7523c3a63244c86ab01"
server_key_iv: "6d0020c22b544328645aa6152b86d557"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.794085 8895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:26.803668 8895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:26.805122 8895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:26.805238 8895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:26.805418 8895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:26.805696 8895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:26.805792 8895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.805828 8895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:26.805851 8895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.810935 8895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35939
I20260924 05:04:26.810992 9023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35939 every 8 connection(s)
I20260924 05:04:26.811273 8895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:26.815718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 8895
I20260924 05:04:26.815819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:26.816000 392 external_mini_cluster.cc:1428] Setting key 0f1b4d71ef8fcd781610490e66ac812b
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:26.819293 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.811860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47251 (local address 127.0.98.62:35395)
0924 05:04:26.811922 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:26.811924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:26.812072 (+ 148us) server_negotiation.cc:407] Connection header received
0924 05:04:26.812876 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:26.812877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:26.812886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:26.812934 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:26.813423 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.813977 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.814839 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.815010 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.815051 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.817047 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:26.817048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:26.817049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:26.817056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:26.818115 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.818452 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.818453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.818453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.818506 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:26.818718 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:26.818718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:26.818719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:26.818825 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:26.818879 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:26.819129 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 05:04:26.819205 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:04:26.819432 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:26.811833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:47251)
0924 05:04:26.811961 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:26.811997 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:26.812802 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:26.812982 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:26.812984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:26.813199 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:26.813364 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:26.813368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.814033 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.814033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:26.814782 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:26.814783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:26.815057 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:26.815058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:26.815148 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:26.815498 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:26.815505 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:26.816962 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:26.818182 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:26.818188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:26.818190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:26.818387 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:26.818564 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:26.818565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:26.818566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:26.818664 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:26.818876 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:26.818881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:26.819006 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 05:04:26.819099 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 05:04:26.819239 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260924 05:04:26.819799 9024 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35395
I20260924 05:04:26.819876 9024 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:26.820036 9024 heartbeater.cc:511] Master 127.0.98.62:35395 requested a full tablet report, sending...
I20260924 05:04:26.820328 8565 ts_manager.cc:194] Registered new tserver with Master: 054a44d5667b47968955afc679594d3c (127.0.98.3:35939)
I20260924 05:04:26.820676 8565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47251
I20260924 05:04:26.827520 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 05:04:26.883920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 8627
I20260924 05:04:26.887390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 8761
I20260924 05:04:26.890825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 8895
I20260924 05:04:26.894515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 8533
I20260924 05:04:26.898407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35395
--webserver_interface=127.0.98.62
--webserver_port=43113
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:26.974436 9033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:26.974624 9033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:26.974645 9033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:26.975704 9033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:26.975734 9033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:26.975747 9033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:26.975759 9033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:26.975771 9033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:26.977141 9033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:26.977314 9033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:26.977465 9033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:04:26.982007 9033 server_base.cc:1034] running on GCE node
W20260924 05:04:26.981974 9038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.981978 9039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:26.981964 9041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:26.982388 9033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:26.982618 9033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:26.983742 9033 hybrid_clock.cc:648] HybridClock initialized: now 1790226266983733 us; error 25 us; skew 500 ppm
Sep 24 05:04:26 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:26.985909 9033 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:26.986250 9033 webserver.cc:460] Webserver started at http://127.0.98.62:43113/ using document root <none> and password file <none>
I20260924 05:04:26.986414 9033 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:26.986460 9033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:26.987587 9033 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:26.988092 9048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.988214 9033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:26.988288 9033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "2c7f05beb2aa4e0bb757f66836de287e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "cc11f7cd4b42fc38761ec36fb9b151df"
server_key_iv: "758143c78ba107618836510dd2912cfe"
server_key_version: "encryptionkey@0"
I20260924 05:04:26.988577 9033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:27.006356 9033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:27.007902 9033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:27.008015 9033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:27.011395 9033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35395
I20260924 05:04:27.011505 9100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35395 every 8 connection(s)
I20260924 05:04:27.011950 9033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:27.012981 9101 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:04:27.013556 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Bootstrap starting.
I20260924 05:04:27.020861 9101 log.cc:824] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:27.022002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9033
I20260924 05:04:27.022248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39751
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35395
--webserver_port=43555
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:04:27.022464 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:04:27.022723 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Bootstrap complete.
I20260924 05:04:27.025104 9101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:27.025318 9101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7f05beb2aa4e0bb757f66836de287e, State: Initialized, Role: FOLLOWER
I20260924 05:04:27.025456 9101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:27.025521 9101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:27.025550 9101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:27.025594 9101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:04:27.026568 9101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:27.026660 9101 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7f05beb2aa4e0bb757f66836de287e; no voters:
I20260924 05:04:27.026808 9101 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:04:27.026856 9105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:04:27.027009 9105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [term 2 LEADER]: Becoming Leader. State: Replica: 2c7f05beb2aa4e0bb757f66836de287e, State: Running, Role: LEADER
I20260924 05:04:27.027057 9101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:27.027196 9105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } }
I20260924 05:04:27.027829 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7f05beb2aa4e0bb757f66836de287e. Latest consensus state: current_term: 2 leader_uuid: "2c7f05beb2aa4e0bb757f66836de287e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } } }
I20260924 05:04:27.027906 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:27.027819 9107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c7f05beb2aa4e0bb757f66836de287e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f05beb2aa4e0bb757f66836de287e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35395 } } }
I20260924 05:04:27.027942 9107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:27.028141 9115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:27.028633 9115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:27.028760 9115 catalog_manager.cc:1259] Loaded cluster ID: 4d9db7be25014a8abdf8265c910daa3d
I20260924 05:04:27.028831 9115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:27.030292 9115 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:27.030484 9115 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c7f05beb2aa4e0bb757f66836de287e: Loaded TSK: 0
I20260924 05:04:27.030723 9115 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:04:27.101090 9104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:27.101275 9104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:27.101296 9104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:27.102443 9104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:27.102483 9104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:27.102509 9104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:27.103868 9104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43555
--tserver_master_addrs=127.0.98.62:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:27.104068 9104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:27.104276 9104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:27.109781 9128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:27.109792 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:27.109786 9127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:27.110013 9104 server_base.cc:1034] running on GCE node
I20260924 05:04:27.110260 9104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:27.110536 9104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:27.111646 9104 hybrid_clock.cc:648] HybridClock initialized: now 1790226267111623 us; error 41 us; skew 500 ppm
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:27.113957 9104 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:04:27.114367 9104 webserver.cc:460] Webserver started at http://127.0.98.1:43555/ using document root <none> and password file <none>
I20260924 05:04:27.114559 9104 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:27.114619 9104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:27.115857 9104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:27.116436 9137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.116622 9104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:27.116703 9104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "3ac999b9e5404ce1bde9e0e425ce716e"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "54895196b49b27667044ea38eb9704df"
server_key_iv: "605410937555e991e973e282645220df"
server_key_version: "encryptionkey@0"
I20260924 05:04:27.117143 9104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:27.134538 9104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:27.136639 9104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:27.136763 9104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:27.136992 9104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:27.137315 9104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:27.137356 9104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.137377 9104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:27.137401 9104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.143486 9104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39751
I20260924 05:04:27.143536 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39751 every 8 connection(s)
I20260924 05:04:27.143855 9104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:27.146770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9104
I20260924 05:04:27.147017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35395
--webserver_port=38219
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.152526 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.144597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:56583)
0924 05:04:27.144750 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.144791 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.145242 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.145655 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.145658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.146237 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.146420 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.146425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.147821 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.147823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.148705 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.148708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.149286 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.149287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.149419 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.149987 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.149997 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.152325 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
W20260924 05:04:27.152541 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.144661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56583 (local address 127.0.98.62:35395)
0924 05:04:27.144834 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.144843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.144876 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:04:27.145424 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.145431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.145486 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.145628 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.146528 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.147727 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.148797 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.149204 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.149280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.152420 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56583: BlockingRecv error: recv got EOF from 127.0.98.1:56583 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
W20260924 05:04:27.152710 9251 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35395 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35395: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.157060 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.152928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:59009)
0924 05:04:27.153031 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.153041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.153092 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.153202 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.153203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.153330 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.153378 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.153380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.153967 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.153968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.154727 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.154730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.155027 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.155028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.155089 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.155405 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.155410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.157017 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 05:04:27.157232 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.152979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59009 (local address 127.0.98.62:35395)
0924 05:04:27.153030 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.153031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.153078 (+ 47us) server_negotiation.cc:407] Connection header received
0924 05:04:27.153146 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.153146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.153155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.153196 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.153435 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.153906 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.154796 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.154962 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.155001 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.157112 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59009: BlockingRecv error: recv got EOF from 127.0.98.1:59009 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.161082 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.157290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:48061)
0924 05:04:27.157326 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.157335 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.157366 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.157481 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.157482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.157574 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.157598 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.157599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.158491 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.158491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.158949 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.158951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.159354 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.159354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.159385 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.159736 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.159739 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.161043 (+ 1304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:04:27.161178 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.157340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48061 (local address 127.0.98.62:35395)
0924 05:04:27.157372 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.157373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.157384 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:27.157418 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.157419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.157428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.157471 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.157654 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.158431 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.159010 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.159302 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.159357 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.161125 (+ 1768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48061: BlockingRecv error: recv got EOF from 127.0.98.1:48061 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:04:27.161180 9251 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:04:27.227313 9255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:27.227559 9255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:27.227591 9255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:27.228816 9255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:27.228857 9255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:27.228892 9255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:27.230326 9255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38219
--tserver_master_addrs=127.0.98.62:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:27.230559 9255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:27.230758 9255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:27.235332 9261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:27.235345 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:27.235391 9255 server_base.cc:1034] running on GCE node
W20260924 05:04:27.235317 9263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:27.235725 9255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:27.235924 9255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:27.237051 9255 hybrid_clock.cc:648] HybridClock initialized: now 1790226267237039 us; error 29 us; skew 500 ppm
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:27.239395 9255 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:04:27.239812 9255 webserver.cc:460] Webserver started at http://127.0.98.2:38219/ using document root <none> and password file <none>
I20260924 05:04:27.239997 9255 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:27.240053 9255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:27.241313 9255 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:27.241849 9270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.241991 9255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.242066 9255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "a9331114d1a5424486b9ef2d5da92df4"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "232ad3fc2f37956dc598f1cea3581426"
server_key_iv: "dd92838acc5047fb6f6e001cc72f89e3"
server_key_version: "encryptionkey@0"
I20260924 05:04:27.242390 9255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:27.265704 9255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:27.267328 9255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:27.267443 9255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:27.267632 9255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:27.267882 9255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:27.267912 9255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.267939 9255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:27.267962 9255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.273073 9255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32899
I20260924 05:04:27.273132 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32899 every 8 connection(s)
I20260924 05:04:27.273406 9255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:04:27.279754 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.274040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43135 (local address 127.0.98.62:35395)
0924 05:04:27.274094 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.274095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.274260 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:04:27.274739 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.274741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.274755 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.274800 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.275647 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.276132 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.277061 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.277258 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.277312 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.279698 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43135: BlockingRecv error: recv got EOF from 127.0.98.2:43135 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:04:27.279794 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.274021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:43135)
0924 05:04:27.274159 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.274191 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.274650 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.274874 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.274877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.275378 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.275560 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.275564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.276209 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.276210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.276980 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.276982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.277322 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.277323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.277431 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.277879 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.277887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.279615 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20260924 05:04:27.280045 9384 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35395 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35395: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 05:04:27.281417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9255
I20260924 05:04:27.281690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35939
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35395
--webserver_port=43587
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.283718 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.280219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:49819)
0924 05:04:27.280256 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.280264 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.280294 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.280416 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.280417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.280509 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.280554 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.280555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.281073 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.281073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.281557 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.281558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.281947 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.281948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.281995 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.282322 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.282326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.283678 (+ 1352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:04:27.283809 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.280262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49819 (local address 127.0.98.62:35395)
0924 05:04:27.280313 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.280315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.280337 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:04:27.280354 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.280355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.280362 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.280408 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.280599 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.281026 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.281621 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.281881 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.281946 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.283757 (+ 1811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49819: BlockingRecv error: recv got EOF from 127.0.98.2:49819 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.287683 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.284034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:36375)
0924 05:04:27.284073 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.284082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.284114 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.284222 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.284223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.284313 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.284334 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.284335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.285102 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.285102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.285551 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.285553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.285991 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.285992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.286030 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.286333 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.286336 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.287646 (+ 1310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:04:27.287784 9384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:04:27.287787 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.284091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36375 (local address 127.0.98.62:35395)
0924 05:04:27.284126 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.284127 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.284149 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:04:27.284166 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.284167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.284174 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.284217 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.284385 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.285030 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.285614 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.285931 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.285997 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.287734 (+ 1737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36375: BlockingRecv error: recv got EOF from 127.0.98.2:36375 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 05:04:27.359342 9387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:27.359519 9387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:27.359539 9387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:27.360630 9387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:27.360663 9387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:27.360687 9387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:27.362010 9387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43587
--tserver_master_addrs=127.0.98.62:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:27.362177 9387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:27.362336 9387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:27.366707 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:27.366706 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:27.366714 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:27.366731 9387 server_base.cc:1034] running on GCE node
I20260924 05:04:27.367097 9387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:27.367312 9387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:27.368443 9387 hybrid_clock.cc:648] HybridClock initialized: now 1790226267368429 us; error 29 us; skew 500 ppm
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:27.370627 9387 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:04:27.371044 9387 webserver.cc:460] Webserver started at http://127.0.98.3:43587/ using document root <none> and password file <none>
I20260924 05:04:27.371248 9387 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:27.371304 9387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:27.372481 9387 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:04:27.372952 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.373090 9387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.373172 9387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "054a44d5667b47968955afc679594d3c"
format_stamp: "Formatted at 2026-09-24 05:04:26 on dist-test-slave-m01t"
server_key: "2531675bc5a5e7523c3a63244c86ab01"
server_key_iv: "6d0020c22b544328645aa6152b86d557"
server_key_version: "encryptionkey@0"
I20260924 05:04:27.373461 9387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:27.387616 9387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:27.389133 9387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:27.389250 9387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:27.389429 9387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:27.389701 9387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:27.389765 9387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.389794 9387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:27.389816 9387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:27.395129 9387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35939
I20260924 05:04:27.395180 9515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35939 every 8 connection(s)
I20260924 05:04:27.395466 9387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:27.395897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9387
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:04:27.403610 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.396053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55453 (local address 127.0.98.62:35395)
0924 05:04:27.396100 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.396102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.396351 (+ 249us) server_negotiation.cc:407] Connection header received
0924 05:04:27.396884 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.396885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.396893 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.396947 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.398089 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.398817 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.400176 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.400442 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.400496 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.403547 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55453: BlockingRecv error: recv got EOF from 127.0.98.3:55453 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:27.403664 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.396059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:55453)
0924 05:04:27.396221 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.396269 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.396800 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.397030 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.397035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.397701 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.397975 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.397982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.398908 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.398909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.400100 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.400103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.400521 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.400523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.400676 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.401024 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.401033 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.403486 (+ 2453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260924 05:04:27.403854 9516 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35395 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35395: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 05:04:27.404443 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.396343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.0.98.62:35395)
0924 05:04:27.396935 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.396936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.396942 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:27.396959 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.396959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.396968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.397025 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.397699 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.398497 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.399548 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.399911 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.399970 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.401384 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:27.401387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:27.401387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:27.401395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:27.403061 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:27.403405 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:27.403407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:27.403407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:27.403485 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:27.403663 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:27.403664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:27.403664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:27.403896 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:27.404031 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:27.404259 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 05:04:27.404356 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":51,"threads_started":1}
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.407568 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.404045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:45837)
0924 05:04:27.404085 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.404093 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.404124 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.404246 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.404246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.404367 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.404417 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.404418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.405062 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.405063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.405599 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.405601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.405930 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.405931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.405974 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.406250 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.406253 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.407530 (+ 1277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 05:04:27.407645 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.404079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45837 (local address 127.0.98.62:35395)
0924 05:04:27.404115 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.404116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.404124 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:27.404181 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.404182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.404190 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.404235 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.404468 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.405008 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.405657 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.405873 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.405926 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.407598 (+ 1672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45837: BlockingRecv error: recv got EOF from 127.0.98.3:45837 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:27 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:27.410876 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:27.407778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:43249)
0924 05:04:27.407807 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:27.407817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:27.407855 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:27.407951 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:27.407951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:27.408038 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:27.408061 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.408062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.408558 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.408558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.409003 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:27.409004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.409248 (+ 244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.409249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:27.409279 (+ 30us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.409523 (+ 244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:27.409526 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:27.410838 (+ 1312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 05:04:27.410943 9254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:27.407823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43249 (local address 127.0.98.62:35395)
0924 05:04:27.407853 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:27.407855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:27.407861 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:27.407903 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:27.407904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:27.407910 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:27.407947 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:27.408103 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.408511 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.409049 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:27.409207 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:27.409244 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:27.410898 (+ 1654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43249: BlockingRecv error: recv got EOF from 127.0.98.3:43249 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:04:27.410987 9516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 05:04:28 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:28 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:28.168591 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:28.161481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:45141)
0924 05:04:28.161618 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:28.161628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:28.161680 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:28.161826 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:28.161827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:28.161969 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:28.162025 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:28.162026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.162612 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.162613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:28.163139 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:28.163141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.163414 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.163414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:28.163448 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:28.166916 (+ 3468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:28.166921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:28.168527 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260924 05:04:28.168677 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:28.161532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45141 (local address 127.0.98.62:35395)
0924 05:04:28.161637 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:28.161638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:28.161667 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:04:28.161750 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:28.161750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:28.161762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:28.161816 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:28.162077 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.162561 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.163195 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.163369 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.163408 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:28.168613 (+ 5205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45141: BlockingRecv error: recv got EOF from 127.0.98.1:45141 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
Sep 24 05:04:28 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:28 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:28.292392 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:28.288169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:51327)
0924 05:04:28.288286 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:28.288295 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:28.288346 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:28.288451 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:28.288452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:28.288600 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:28.288655 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:28.288657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.289247 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.289248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:28.289814 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:28.289817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.290119 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.290120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:28.290156 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:28.290625 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:28.290630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:28.292324 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
W20260924 05:04:28.292475 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:28.288222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51327 (local address 127.0.98.62:35395)
0924 05:04:28.288267 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:28.288269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:28.288347 (+ 78us) server_negotiation.cc:407] Connection header received
0924 05:04:28.288383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:28.288384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:28.288395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:28.288439 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:28.288707 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.289194 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.289875 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.290069 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.290117 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:28.292417 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51327: BlockingRecv error: recv got EOF from 127.0.98.2:51327 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 05:04:28 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:28 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:28.415307 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:28.411307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:45599)
0924 05:04:28.411427 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:28.411437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:28.411491 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:28.411598 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:28.411599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:28.411747 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:28.411792 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:28.411794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.412388 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.412388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:28.412928 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:28.412930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.413193 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.413193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:28.413227 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:28.413645 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:28.413650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:28.415246 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260924 05:04:28.415381 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:28.411348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45599 (local address 127.0.98.62:35395)
0924 05:04:28.411389 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:28.411391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:28.411486 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:04:28.411540 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:28.411540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:28.411553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:28.411588 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:28.411839 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.412334 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.412979 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:28.413150 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:28.413189 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:28.415325 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45599: BlockingRecv error: recv got EOF from 127.0.98.3:45599 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 05:04:29 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:29 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:29.173116 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:29.168997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:53983)
0924 05:04:29.169108 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:29.169119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:29.169168 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:29.169274 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:29.169275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:29.169415 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:29.169462 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:29.169463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.170075 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.170076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:29.170578 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:29.170580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.170862 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.170863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:29.170898 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:29.171328 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:29.171333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:29.173040 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
W20260924 05:04:29.173190 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:29.169060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53983 (local address 127.0.98.62:35395)
0924 05:04:29.169154 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:29.169156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:29.169164 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:29.169209 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:29.169209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:29.169218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:29.169270 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:29.169507 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.170026 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.170633 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.170813 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.170854 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:29.173130 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53983: BlockingRecv error: recv got EOF from 127.0.98.1:53983 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
Sep 24 05:04:29 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:29 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:29.296649 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:29.292808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:51913)
0924 05:04:29.292947 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:29.292957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:29.293016 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:29.293124 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:29.293125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:29.293273 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:29.293319 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:29.293321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.293932 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.293933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:29.294434 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:29.294435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.294701 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.294702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:29.294736 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:29.295116 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:29.295119 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:29.296587 (+ 1468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
W20260924 05:04:29.296731 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:29.292863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51913 (local address 127.0.98.62:35395)
0924 05:04:29.292920 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:29.292922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:29.293017 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:04:29.293054 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:29.293055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:29.293066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:29.293110 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:29.293367 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.293884 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.294485 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.294657 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.294697 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:29.296674 (+ 1977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51913: BlockingRecv error: recv got EOF from 127.0.98.2:51913 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 05:04:29 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:29 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:29.419539 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:29.415660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:52843)
0924 05:04:29.415764 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:29.415775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:29.415828 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:29.415925 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:29.415926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:29.416062 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:29.416109 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:29.416110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.416677 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.416677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:29.417184 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:29.417185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.417444 (+ 259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.417444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:29.417477 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:29.417879 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:29.417883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:29.419466 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
W20260924 05:04:29.419616 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:29.415756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52843 (local address 127.0.98.62:35395)
0924 05:04:29.415792 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:29.415794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:29.415815 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:04:29.415865 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:29.415865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:29.415875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:29.415914 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:29.416153 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.416627 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.417233 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:29.417399 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:29.417438 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:29.419546 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52843: BlockingRecv error: recv got EOF from 127.0.98.3:52843 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 05:04:30 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:30 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:30.177736 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:30.173563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:47151)
0924 05:04:30.173693 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:30.173702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:30.173760 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:30.173887 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:30.173888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:30.174029 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:30.174076 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:30.174077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.174679 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.174679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:30.175187 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:30.175189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.175456 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.175457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:30.175490 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:30.175903 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:30.175908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:30.177643 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260924 05:04:30.177794 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:30.173609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47151 (local address 127.0.98.62:35395)
0924 05:04:30.173743 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:30.173745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:30.173755 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:30.173804 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:30.173805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:30.173818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:30.173870 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:30.174129 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.174630 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.175237 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.175412 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.175452 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:30.177742 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47151: BlockingRecv error: recv got EOF from 127.0.98.1:47151 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
Sep 24 05:04:30 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:30 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:30.300926 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:30.297064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:54473)
0924 05:04:30.297229 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:30.297238 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:30.297291 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:30.297405 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:30.297405 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:30.297555 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:30.297609 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:30.297611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.298213 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.298213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:30.298724 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:30.298726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.298982 (+ 256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.298983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:30.299017 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:30.299382 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:30.299386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:30.300866 (+ 1480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
W20260924 05:04:30.301017 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:30.297141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54473 (local address 127.0.98.62:35395)
0924 05:04:30.297207 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:30.297209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:30.297288 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:04:30.297329 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:30.297330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:30.297346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:30.297393 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:30.297657 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.298164 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.298774 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.298939 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.298979 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:30.300951 (+ 1972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54473: BlockingRecv error: recv got EOF from 127.0.98.2:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 05:04:30 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:30 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:30.424103 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:30.419936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:39903)
0924 05:04:30.420086 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:30.420096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:30.420152 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:30.420253 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:30.420254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:30.420417 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:30.420466 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:30.420467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.421077 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.421078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:30.421609 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:30.421611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.421934 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.421935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:30.421974 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:30.422423 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:30.422427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:30.424044 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
W20260924 05:04:30.424201 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:30.419985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39903 (local address 127.0.98.62:35395)
0924 05:04:30.420039 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:30.420040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:30.420135 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:04:30.420190 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:30.420191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:30.420202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:30.420247 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:30.420511 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.421013 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.421672 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:30.421878 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:30.421922 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:30.424132 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39903: BlockingRecv error: recv got EOF from 127.0.98.3:39903 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 05:04:31 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:31 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:31.182240 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:31.178140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:41291)
0924 05:04:31.178259 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:31.178268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:31.178314 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:31.178431 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:31.178431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:31.178573 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:31.178624 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:31.178626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.179205 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.179206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:31.179729 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:31.179730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.179997 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.179997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:31.180030 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:31.180408 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:31.180412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:31.182173 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
W20260924 05:04:31.182322 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:31.178187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41291 (local address 127.0.98.62:35395)
0924 05:04:31.178300 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:31.178301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:31.178309 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:31.178355 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:31.178356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:31.178365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:31.178421 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:31.178670 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.179155 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.179777 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.179954 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.179993 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:31.182258 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41291: BlockingRecv error: recv got EOF from 127.0.98.1:41291 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
Sep 24 05:04:31 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:31 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:31.305207 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:31.301323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:52063)
0924 05:04:31.301435 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:31.301445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:31.301496 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:31.301603 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:31.301604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:31.301778 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:31.301823 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:31.301825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.302402 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.302402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:31.302916 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:31.302917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.303187 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.303187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:31.303221 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:31.303608 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:31.303612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:31.305148 (+ 1536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
W20260924 05:04:31.305289 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:31.301385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52063 (local address 127.0.98.62:35395)
0924 05:04:31.301438 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:31.301440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:31.301486 (+ 46us) server_negotiation.cc:407] Connection header received
0924 05:04:31.301552 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:31.301553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:31.301562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:31.301597 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:31.301867 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.302352 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.302965 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.303142 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.303181 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:31.305232 (+ 2051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52063: BlockingRecv error: recv got EOF from 127.0.98.2:52063 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 05:04:31 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:31 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:31.428785 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:31.424503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:42869)
0924 05:04:31.424641 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:31.424651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:31.424709 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:31.424813 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:31.424814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:31.424979 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:31.425028 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:31.425030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.425621 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.425622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:31.426194 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:31.426197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.426485 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.426486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:31.426523 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:31.426933 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:31.426937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:31.428716 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260924 05:04:31.428844 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:31.424559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42869 (local address 127.0.98.62:35395)
0924 05:04:31.424612 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:31.424614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:31.424701 (+ 87us) server_negotiation.cc:407] Connection header received
0924 05:04:31.424748 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:31.424749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:31.424760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:31.424803 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:31.425077 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.425562 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.426259 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:31.426437 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:31.426479 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:31.428793 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42869: BlockingRecv error: recv got EOF from 127.0.98.3:42869 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 05:04:32 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:32 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:32.186753 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:32.182632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:34125)
0924 05:04:32.182746 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:32.182758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:32.182810 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:32.182932 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:32.182933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:32.183085 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:32.183132 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:32.183134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.183768 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.183769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:32.184394 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:32.184397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.184695 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.184695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:32.184729 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:32.185103 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:32.185107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:32.186680 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
W20260924 05:04:32.186825 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:32.182680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34125 (local address 127.0.98.62:35395)
0924 05:04:32.182800 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:32.182802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:32.182810 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:32.182860 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:32.182861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:32.182871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:32.182923 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:32.183182 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.183700 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.184459 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.184645 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.184692 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:32.186766 (+ 2074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34125: BlockingRecv error: recv got EOF from 127.0.98.1:34125 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
Sep 24 05:04:32 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:32 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:32.309638 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:32.305622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:58967)
0924 05:04:32.305754 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:32.305766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:32.305825 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:32.305948 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:32.305948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:32.306107 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:32.306152 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:32.306154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.306765 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.306766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:32.307282 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:32.307283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.307595 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.307596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:32.307650 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:32.308026 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:32.308030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:32.309574 (+ 1544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":52,"threads_started":1}
W20260924 05:04:32.309741 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:32.305739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58967 (local address 127.0.98.62:35395)
0924 05:04:32.305785 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:32.305787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:32.305808 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:04:32.305882 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:32.305882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:32.305894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:32.305938 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:32.306218 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.306696 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.307353 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.307538 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.307591 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:32.309658 (+ 2067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58967: BlockingRecv error: recv got EOF from 127.0.98.2:58967 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 05:04:32 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:32 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:32.433483 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:32.429193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:36049)
0924 05:04:32.429314 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:32.429326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:32.429400 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:32.429531 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:32.429532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:32.429702 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:32.429764 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:32.429766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.430405 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.430407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:32.430933 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:32.430935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.431267 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.431267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:32.431322 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:32.431767 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:32.431771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:32.433406 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
W20260924 05:04:32.433561 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:32.429254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36049 (local address 127.0.98.62:35395)
0924 05:04:32.429318 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:32.429320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:32.429389 (+ 69us) server_negotiation.cc:407] Connection header received
0924 05:04:32.429456 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:32.429457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:32.429471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:32.429521 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:32.429837 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.430332 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.431013 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:32.431208 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:32.431261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:32.433507 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36049: BlockingRecv error: recv got EOF from 127.0.98.3:36049 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 05:04:33 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:33 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:33.196051 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:33.187151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:49541)
0924 05:04:33.187262 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:33.187273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:33.187326 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:33.187437 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:33.187438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:33.187577 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:33.187622 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:33.187624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.188216 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.188217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:33.188725 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:33.188726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.189003 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.189004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:33.189038 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:33.194396 (+ 5358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:33.194401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:33.195992 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
W20260924 05:04:33.196152 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:33.187183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49541 (local address 127.0.98.62:35395)
0924 05:04:33.187298 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:33.187299 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:33.187310 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:33.187373 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:33.187373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:33.187383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:33.187435 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:33.187669 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.188164 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.188777 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.188957 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.188999 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:33.196085 (+ 7086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49541: BlockingRecv error: recv got EOF from 127.0.98.1:49541 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
Sep 24 05:04:33 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:33 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:33.314081 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:33.310045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:44427)
0924 05:04:33.310175 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:33.310187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:33.310243 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:33.310349 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:33.310349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:33.310501 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:33.310554 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:33.310556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.311151 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.311152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:33.311679 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:33.311681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.311946 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.311946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:33.311981 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:33.312344 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:33.312348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:33.314000 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
W20260924 05:04:33.314122 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:33.310107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44427 (local address 127.0.98.62:35395)
0924 05:04:33.310164 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:33.310166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:33.310237 (+ 71us) server_negotiation.cc:407] Connection header received
0924 05:04:33.310283 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:33.310284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:33.310295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:33.310338 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:33.310601 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.311091 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.311732 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.311901 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.311940 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:33.314069 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44427: BlockingRecv error: recv got EOF from 127.0.98.2:44427 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 05:04:33 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:33 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:33.438284 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:33.433949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:52935)
0924 05:04:33.434086 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:33.434100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:33.434171 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:33.434290 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:33.434291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:33.434467 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:33.434513 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:33.434515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.435158 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.435159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:33.435708 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:33.435710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.436040 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.436041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:33.436093 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:33.436555 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:33.436560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:33.438225 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260924 05:04:33.438372 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:33.434030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52935 (local address 127.0.98.62:35395)
0924 05:04:33.434108 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:33.434110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:33.434151 (+ 41us) server_negotiation.cc:407] Connection header received
0924 05:04:33.434221 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:33.434222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:33.434235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:33.434284 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:33.434591 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.435087 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.435788 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:33.435982 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:33.436035 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:33.438309 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52935: BlockingRecv error: recv got EOF from 127.0.98.3:52935 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 05:04:34 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:34 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:34.200282 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:34.196477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:54097)
0924 05:04:34.196591 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:34.196601 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:34.196645 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:34.196756 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:34.196757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:34.196903 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:34.196949 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:34.196950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.197537 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.197537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:34.198055 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:34.198056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.198337 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.198338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:34.198371 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:34.198735 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:34.198739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:34.200228 (+ 1489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
W20260924 05:04:34.200388 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:34.196537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54097 (local address 127.0.98.62:35395)
0924 05:04:34.196655 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:34.196656 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:34.196663 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:34.196682 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:34.196682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:34.196692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:34.196747 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:34.196996 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.197487 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.198114 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.198292 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.198336 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:34.200310 (+ 1974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54097: BlockingRecv error: recv got EOF from 127.0.98.1:54097 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
Sep 24 05:04:34 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:34 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:34.318475 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:34.314511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:51909)
0924 05:04:34.314640 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:34.314650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:34.314712 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:34.314827 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:34.314827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:34.314969 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:34.315015 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:34.315016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.315609 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.315609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:34.316110 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:34.316111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.316366 (+ 255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.316367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:34.316401 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:34.316773 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:34.316776 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:34.318397 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
W20260924 05:04:34.318538 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:34.314575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51909 (local address 127.0.98.62:35395)
0924 05:04:34.314644 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:34.314646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:34.314703 (+ 57us) server_negotiation.cc:407] Connection header received
0924 05:04:34.314757 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:34.314758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:34.314768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:34.314813 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:34.315062 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.315552 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.316156 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.316324 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.316363 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:34.318485 (+ 2122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51909: BlockingRecv error: recv got EOF from 127.0.98.2:51909 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 05:04:34 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:34 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:34.442906 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:34.438679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:37451)
0924 05:04:34.438802 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:34.438813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:34.438873 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:34.438990 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:34.438991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:34.439158 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:34.439204 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:34.439205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.439835 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.439836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:34.440357 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:34.440359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.440694 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.440695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:34.440751 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:34.441199 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:34.441203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:34.442829 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260924 05:04:34.442973 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:34.438729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37451 (local address 127.0.98.62:35395)
0924 05:04:34.438774 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:34.438776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:34.438867 (+ 91us) server_negotiation.cc:407] Connection header received
0924 05:04:34.438921 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:34.438922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:34.438936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:34.438984 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:34.439275 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.439764 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.440449 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:34.440635 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:34.440689 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:34.442922 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37451: BlockingRecv error: recv got EOF from 127.0.98.3:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 05:04:35 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:35 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:35.204682 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:35.200697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:36159)
0924 05:04:35.200831 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:35.200840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:35.200885 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:35.201013 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:35.201013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:35.201165 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:35.201210 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:35.201213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.201817 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.201818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:35.202326 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:35.202328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.202594 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.202595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:35.202628 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:35.202993 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:35.202996 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:35.204608 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260924 05:04:35.204758 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:35.200801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36159 (local address 127.0.98.62:35395)
0924 05:04:35.200908 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:35.200909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:35.200916 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:35.200936 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:35.200936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:35.200947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:35.201010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:35.201259 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.201771 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.202377 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.202552 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.202591 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:35.204699 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36159: BlockingRecv error: recv got EOF from 127.0.98.1:36159 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
Sep 24 05:04:35 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:35 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:35.323279 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:35.318902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:38775)
0924 05:04:35.319031 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:35.319043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:35.319104 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:35.319256 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:35.319257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:35.319435 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:35.319481 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:35.319484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.320249 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.320251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:35.320786 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:35.320787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.321114 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.321115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:35.321167 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:35.321558 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:35.321562 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:35.323220 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260924 05:04:35.323401 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:35.318968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38775 (local address 127.0.98.62:35395)
0924 05:04:35.319028 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:35.319030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:35.319102 (+ 72us) server_negotiation.cc:407] Connection header received
0924 05:04:35.319164 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:35.319165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:35.319176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:35.319248 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:35.319550 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.320177 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.320860 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.321048 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.321110 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:35.323330 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38775: BlockingRecv error: recv got EOF from 127.0.98.2:38775 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 05:04:35 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:35 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:35.447184 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:35.443291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:43633)
0924 05:04:35.443435 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:35.443445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:35.443509 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:35.443610 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:35.443611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:35.443753 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:35.443797 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:35.443798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.444370 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.444371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:35.444918 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:35.444920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.445212 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.445213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:35.445247 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:35.445626 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:35.445630 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:35.447128 (+ 1498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
W20260924 05:04:35.447258 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:35.443342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43633 (local address 127.0.98.62:35395)
0924 05:04:35.443393 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:35.443395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:35.443500 (+ 105us) server_negotiation.cc:407] Connection header received
0924 05:04:35.443548 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:35.443549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:35.443561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:35.443606 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:35.443841 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.444320 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.444981 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:35.445166 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:35.445208 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:35.447201 (+ 1993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43633: BlockingRecv error: recv got EOF from 127.0.98.3:43633 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 05:04:36 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:36 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:36.209048 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:36.205087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:37239)
0924 05:04:36.205219 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:36.205229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:36.205286 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:36.205395 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:36.205396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:36.205546 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:36.205591 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:36.205592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.206202 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.206203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:36.206703 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:36.206705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.206977 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.206977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:36.207010 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:36.207401 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:36.207405 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:36.208993 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
W20260924 05:04:36.209156 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:36.205145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37239 (local address 127.0.98.62:35395)
0924 05:04:36.205243 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:36.205245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:36.205270 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:04:36.205325 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:36.205325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:36.205335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:36.205386 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:36.205640 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.206155 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.206754 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.206934 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.206973 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:36.209079 (+ 2106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37239: BlockingRecv error: recv got EOF from 127.0.98.1:37239 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":54,"threads_started":1}
Sep 24 05:04:36 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:36 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:36.328040 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:36.323662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:58827)
0924 05:04:36.323802 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:36.323815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:36.323884 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:36.324002 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:36.324002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:36.324184 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:36.324231 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:36.324233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.324859 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.324860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:36.325389 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:36.325391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.325739 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.325740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:36.325798 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:36.326211 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:36.326216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:36.327977 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
W20260924 05:04:36.328125 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:36.323740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58827 (local address 127.0.98.62:35395)
0924 05:04:36.323802 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:36.323804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:36.323865 (+ 61us) server_negotiation.cc:407] Connection header received
0924 05:04:36.323934 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:36.323935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:36.323949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:36.323992 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:36.324301 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.324791 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.325463 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.325655 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.325735 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:36.328063 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58827: BlockingRecv error: recv got EOF from 127.0.98.2:58827 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 05:04:36 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:36 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:36.451905 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:36.447643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:43589)
0924 05:04:36.447786 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:36.447799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:36.447870 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:36.448010 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:36.448011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:36.448185 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:36.448230 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:36.448232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.448867 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.448868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:36.449397 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:36.449398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.449706 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.449707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:36.449787 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:36.450188 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:36.450193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:36.451828 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
W20260924 05:04:36.451972 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:36.447735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43589 (local address 127.0.98.62:35395)
0924 05:04:36.447791 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:36.447793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:36.447849 (+ 56us) server_negotiation.cc:407] Connection header received
0924 05:04:36.447930 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:36.447931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:36.447950 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:36.447994 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:36.448297 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.448795 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.449468 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:36.449649 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:36.449702 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:36.451917 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43589: BlockingRecv error: recv got EOF from 127.0.98.3:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 05:04:37 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:37 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:37.213423 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:37.209437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:47967)
0924 05:04:37.209576 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:37.209587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:37.209641 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:37.209792 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:37.209793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:37.209951 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:37.209996 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:37.209998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.210605 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.210606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:37.211120 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:37.211121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.211427 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.211428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:37.211470 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:37.211870 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:37.211874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:37.213363 (+ 1489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
W20260924 05:04:37.213502 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:37.209541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47967 (local address 127.0.98.62:35395)
0924 05:04:37.209645 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:37.209646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:37.209670 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:04:37.209689 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:37.209689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:37.209698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:37.209760 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:37.210060 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.210544 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.211175 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.211348 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.211387 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:37.213444 (+ 2057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47967: BlockingRecv error: recv got EOF from 127.0.98.1:47967 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":42,"threads_started":1}
Sep 24 05:04:37 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:37 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:37.332631 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:37.328476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:52865)
0924 05:04:37.328611 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:37.328622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:37.328688 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:37.328816 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:37.328817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:37.328993 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:37.329040 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:37.329042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.329683 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.329684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:37.330219 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:37.330221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.330560 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.330561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:37.330617 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:37.331005 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:37.331008 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:37.332571 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
W20260924 05:04:37.332719 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:37.328530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52865 (local address 127.0.98.62:35395)
0924 05:04:37.328591 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:37.328593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:37.328680 (+ 87us) server_negotiation.cc:407] Connection header received
0924 05:04:37.328741 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:37.328741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:37.328755 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:37.328797 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:37.329113 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.329609 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.330296 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.330498 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.330554 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:37.332661 (+ 2107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52865: BlockingRecv error: recv got EOF from 127.0.98.2:52865 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 05:04:37 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:37 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:37.456327 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:37.452327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:57963)
0924 05:04:37.452460 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:37.452471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:37.452533 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:37.452651 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:37.452652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:37.452799 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:37.452846 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:37.452848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.453450 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.453451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:37.454040 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:37.454042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.454320 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.454320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:37.454356 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:37.454737 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:37.454741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:37.456261 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
W20260924 05:04:37.456390 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:37.452424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57963 (local address 127.0.98.62:35395)
0924 05:04:37.452484 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:37.452485 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:37.452528 (+ 43us) server_negotiation.cc:407] Connection header received
0924 05:04:37.452581 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:37.452581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:37.452600 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:37.452635 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:37.452892 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.453365 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.454099 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:37.454271 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:37.454315 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:37.456339 (+ 2024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57963: BlockingRecv error: recv got EOF from 127.0.98.3:57963 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 05:04:38 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:38 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:38.223626 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:38.213848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:51793)
0924 05:04:38.213963 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:38.213972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:38.214018 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:38.214135 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:38.214135 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:38.214282 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:38.214330 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:38.214332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.214913 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.214914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:38.215417 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:38.215418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.215691 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.215691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:38.215727 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:38.221918 (+ 6191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:38.221924 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:38.223563 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
W20260924 05:04:38.223726 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:38.213906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51793 (local address 127.0.98.62:35395)
0924 05:04:38.214019 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:38.214021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:38.214047 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:04:38.214065 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:38.214066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:38.214075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:38.214121 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:38.214381 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.214864 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.215470 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.215645 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.215684 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:38.223657 (+ 7973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51793: BlockingRecv error: recv got EOF from 127.0.98.1:51793 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
Sep 24 05:04:38 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:38 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:38.337322 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:38.333098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:55999)
0924 05:04:38.333249 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:38.333260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:38.333332 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:38.333460 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:38.333461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:38.333645 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:38.333690 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:38.333692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.334345 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.334346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:38.334875 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:38.334877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.335202 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.335202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:38.335258 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:38.335680 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:38.335685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:38.337246 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
W20260924 05:04:38.337399 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:38.333163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55999 (local address 127.0.98.62:35395)
0924 05:04:38.333222 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:38.333224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:38.333323 (+ 99us) server_negotiation.cc:407] Connection header received
0924 05:04:38.333388 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:38.333389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:38.333404 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:38.333451 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:38.333776 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.334275 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.334952 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.335143 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.335196 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:38.337347 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55999: BlockingRecv error: recv got EOF from 127.0.98.2:55999 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 05:04:38 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:38 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:38.460876 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:38.456717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:58733)
0924 05:04:38.456844 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:38.456855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:38.456926 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:38.457058 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:38.457058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:38.457219 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:38.457264 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:38.457266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.457941 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.457942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:38.458459 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:38.458461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.458788 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.458789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:38.458840 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:38.459227 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:38.459231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:38.460809 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
W20260924 05:04:38.460947 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:38.456777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58733 (local address 127.0.98.62:35395)
0924 05:04:38.456837 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:38.456839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:38.456918 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:04:38.456976 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:38.456976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:38.456990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:38.457043 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:38.457333 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.457869 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.458531 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:38.458732 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:38.458786 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:38.460889 (+ 2103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58733: BlockingRecv error: recv got EOF from 127.0.98.3:58733 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 05:04:39 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:39 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:39.228322 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:39.224078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:49829)
0924 05:04:39.224231 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:39.224244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:39.224299 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:39.224426 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:39.224427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:39.224574 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:39.224622 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:39.224624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.225223 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.225224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:39.225772 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:39.225774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.226068 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.226068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:39.226110 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:39.226542 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:39.226547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:39.228261 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260924 05:04:39.228434 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:39.224154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49829 (local address 127.0.98.62:35395)
0924 05:04:39.224272 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:39.224273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:39.224286 (+ 13us) server_negotiation.cc:407] Connection header received
0924 05:04:39.224356 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:39.224357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:39.224367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:39.224412 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:39.224672 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.225170 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.225830 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.226016 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.226063 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:39.228365 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49829: BlockingRecv error: recv got EOF from 127.0.98.1:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
Sep 24 05:04:39 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:39 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:39.341894 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:39.337825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:45455)
0924 05:04:39.337971 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:39.337982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:39.338041 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:39.338168 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:39.338169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:39.338328 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:39.338373 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:39.338374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.338953 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.338953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:39.339459 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:39.339460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.339739 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.339740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:39.339776 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:39.340168 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:39.340172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:39.341825 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
W20260924 05:04:39.341979 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:39.337912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45455 (local address 127.0.98.62:35395)
0924 05:04:39.337966 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:39.337968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:39.338036 (+ 68us) server_negotiation.cc:407] Connection header received
0924 05:04:39.338104 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:39.338105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:39.338117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:39.338155 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:39.338420 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.338902 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.339520 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.339694 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.339734 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:39.341915 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45455: BlockingRecv error: recv got EOF from 127.0.98.2:45455 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 05:04:39 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:39 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:39.465194 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:39.461261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:53293)
0924 05:04:39.461372 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:39.461382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:39.461436 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:39.461536 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:39.461536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:39.461686 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:39.461747 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:39.461749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.462322 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.462323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:39.462857 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:39.462858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.463124 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.463125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:39.463159 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:39.463570 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:39.463574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:39.465122 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
W20260924 05:04:39.465247 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:39.461315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53293 (local address 127.0.98.62:35395)
0924 05:04:39.461378 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:39.461380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:39.461430 (+ 50us) server_negotiation.cc:407] Connection header received
0924 05:04:39.461483 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:39.461484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:39.461496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:39.461530 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:39.461792 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.462272 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.462906 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:39.463080 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:39.463119 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:39.465200 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53293: BlockingRecv error: recv got EOF from 127.0.98.3:53293 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 05:04:40 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:40 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:40.233196 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:40.228950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:55583)
0924 05:04:40.229096 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:40.229109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:40.229179 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:40.229312 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:40.229313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:40.229512 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:40.229560 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:40.229563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.230206 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.230207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:40.230720 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:40.230722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.231012 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.231013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:40.231047 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:40.231489 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:40.231494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:40.233122 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260924 05:04:40.233290 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:40.229004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55583 (local address 127.0.98.62:35395)
0924 05:04:40.229108 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:40.229111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:40.229174 (+ 63us) server_negotiation.cc:407] Connection header received
0924 05:04:40.229230 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:40.229230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:40.229244 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:40.229307 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:40.229629 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.230149 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.230781 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.230963 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.231009 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:40.233215 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55583: BlockingRecv error: recv got EOF from 127.0.98.1:55583 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":45,"threads_started":1}
Sep 24 05:04:40 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:40 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:40.346447 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:40.342347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:44071)
0924 05:04:40.342485 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:40.342496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:40.342554 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:40.342671 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:40.342672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:40.342820 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:40.342867 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:40.342869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.343434 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.343434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:40.343937 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:40.343939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.344201 (+ 262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.344201 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:40.344236 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:40.344652 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:40.344656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:40.346390 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
W20260924 05:04:40.346529 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:40.342454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44071 (local address 127.0.98.62:35395)
0924 05:04:40.342511 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:40.342513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:40.342538 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:04:40.342602 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:40.342603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:40.342613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:40.342660 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:40.342912 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.343384 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.343987 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.344155 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.344195 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:40.346472 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44071: BlockingRecv error: recv got EOF from 127.0.98.2:44071 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 05:04:40 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:40 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:40.469475 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:40.465561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:54203)
0924 05:04:40.465681 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:40.465691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:40.465780 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:40.465892 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:40.465893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:40.466048 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:40.466096 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:40.466098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.466672 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.466672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:40.467186 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:40.467187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.467443 (+ 256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.467444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:40.467477 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:40.467849 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:40.467852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:40.469406 (+ 1554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
W20260924 05:04:40.469553 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:40.465624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54203 (local address 127.0.98.62:35395)
0924 05:04:40.465691 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:40.465693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:40.465775 (+ 82us) server_negotiation.cc:407] Connection header received
0924 05:04:40.465820 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:40.465821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:40.465833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:40.465877 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:40.466145 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.466622 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.467236 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:40.467400 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:40.467439 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:40.469482 (+ 2043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54203: BlockingRecv error: recv got EOF from 127.0.98.3:54203 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 05:04:41 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:41 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:41.238243 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:41.233675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:51469)
0924 05:04:41.233807 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:41.233817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:41.233867 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:41.234010 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:41.234010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:41.234173 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:41.234221 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:41.234223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.235003 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.235004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:41.235594 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:41.235597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.235919 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.235920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:41.235965 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:41.236417 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:41.236422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:41.238149 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260924 05:04:41.238312 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:41.233771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51469 (local address 127.0.98.62:35395)
0924 05:04:41.233887 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:41.233889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:41.233896 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:04:41.233921 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:41.233922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:41.233933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:41.234001 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:41.234318 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.234923 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.235662 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.235852 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.235901 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:41.238250 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51469: BlockingRecv error: recv got EOF from 127.0.98.1:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Sep 24 05:04:41 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:41 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:41.350790 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:41.346862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:44109)
0924 05:04:41.346972 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:41.346983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:41.347035 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:41.347144 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:41.347145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:41.347293 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:41.347337 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:41.347339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.347910 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.347910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:41.348409 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:41.348410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.348675 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.348675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:41.348710 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:41.349089 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:41.349093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:41.350704 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
W20260924 05:04:41.350895 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:41.346927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44109 (local address 127.0.98.62:35395)
0924 05:04:41.347030 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:41.347032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:41.347040 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:04:41.347081 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:41.347081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:41.347092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:41.347136 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:41.347385 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.347860 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.348460 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.348629 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.348669 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:41.350800 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44109: BlockingRecv error: recv got EOF from 127.0.98.2:44109 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 05:04:41 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:41 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:41.473891 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:41.469895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.3:35153)
0924 05:04:41.470027 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:41.470037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:41.470098 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:41.470206 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:41.470207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:41.470353 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:41.470408 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:41.470409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.470980 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.470980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:41.471506 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:41.471507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.471781 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.471782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:41.471828 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:41.472247 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:41.472251 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:41.473821 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
W20260924 05:04:41.473948 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:41.469991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35153 (local address 127.0.98.62:35395)
0924 05:04:41.470034 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:41.470036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:41.470098 (+ 62us) server_negotiation.cc:407] Connection header received
0924 05:04:41.470138 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:41.470139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:41.470151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:41.470193 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:41.470453 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.470929 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.471560 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:41.471731 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:41.471770 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:41.473898 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35153: BlockingRecv error: recv got EOF from 127.0.98.3:35153 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 05:04:42 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:42 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:42.242944 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.238794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.1:52857)
0924 05:04:42.238924 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:42.238936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:42.238985 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:42.239122 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:42.239123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:42.239268 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:42.239315 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.239316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.239985 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.239986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.240623 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.240626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.240933 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.240933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.240967 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.241343 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:42.241347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:42.242886 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
W20260924 05:04:42.243055 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:42.238884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52857 (local address 127.0.98.62:35395)
0924 05:04:42.239002 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.239003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.239012 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:42.239035 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.239036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.239046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.239112 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.239368 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.239910 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.240688 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.240882 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.240931 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.242975 (+ 2044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52857: BlockingRecv error: recv got EOF from 127.0.98.1:52857 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
Sep 24 05:04:42 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:04:42 dist-test-slave-m01t krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:04:42.355161 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.351241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35395 (local address 127.0.98.2:39523)
0924 05:04:42.351375 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:42.351386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:42.351440 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:42.351546 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:42.351547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:42.351697 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:42.351742 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.351744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.352337 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.352337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.352843 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.352844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.353119 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.353119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.353153 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.353539 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:42.353543 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:42.355104 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
W20260924 05:04:42.355242 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:42.351291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39523 (local address 127.0.98.62:35395)
0924 05:04:42.351346 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.351348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.351437 (+ 89us) server_negotiation.cc:407] Connection header received
0924 05:04:42.351483 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.351484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.351494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.351540 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.351788 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.352283 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.352895 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.353073 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.353117 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.355185 (+ 2068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39523: BlockingRecv error: recv got EOF from 127.0.98.2:39523 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:04:42.398183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9104
I20260924 05:04:42.401885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9255
I20260924 05:04:42.405300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9387
I20260924 05:04:42.408804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9033
2026-09-24T05:04:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17219 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T05:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:42Z Disabled control of system clock
I20260924 05:04:42.433684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35131
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 05:04:42.511173 9583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:42.511353 9583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:42.511371 9583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:42.512429 9583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:42.512463 9583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:42.512475 9583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:42.512493 9583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:42.512511 9583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:42.513764 9583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:42.513937 9583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:42.514159 9583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:42.518565 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:42.518606 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.518649 9583 server_base.cc:1034] running on GCE node
W20260924 05:04:42.518620 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.518945 9583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:42.519198 9583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:42.520339 9583 hybrid_clock.cc:648] HybridClock initialized: now 1790226282520324 us; error 27 us; skew 500 ppm
I20260924 05:04:42.521466 9583 webserver.cc:460] Webserver started at http://127.0.98.62:36037/ using document root <none> and password file <none>
I20260924 05:04:42.521698 9583 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:42.521782 9583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:42.521906 9583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:42.522905 9583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "e13378c4ccd042e898c389b5e44a3864"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "35958f00dd72bf64a12fbbf3a39a6363"
server_key_iv: "585f4bb84303e97ee6243ec1e2a62cc2"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.523231 9583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "e13378c4ccd042e898c389b5e44a3864"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "35958f00dd72bf64a12fbbf3a39a6363"
server_key_iv: "585f4bb84303e97ee6243ec1e2a62cc2"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.524360 9583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:42.524968 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.525120 9583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:42.525199 9583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e13378c4ccd042e898c389b5e44a3864"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "35958f00dd72bf64a12fbbf3a39a6363"
server_key_iv: "585f4bb84303e97ee6243ec1e2a62cc2"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.525264 9583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:42.538914 9583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:42.539232 9583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:42.539366 9583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:42.543077 9583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35131
I20260924 05:04:42.543111 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35131 every 8 connection(s)
I20260924 05:04:42.543426 9583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:42.544065 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:42.546597 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864: Bootstrap starting.
I20260924 05:04:42.547127 9650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:42.547300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9583
I20260924 05:04:42.547380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:42.547384 9650 log.cc:824] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:42.547561 392 external_mini_cluster.cc:1428] Setting key 1fbfa52af758954e8b0591d989b04949
I20260924 05:04:42.548054 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864: No bootstrap required, opened a new log
I20260924 05:04:42.550551 9650 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13378c4ccd042e898c389b5e44a3864" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35131 } }
I20260924 05:04:42.550710 9650 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:42.550740 9650 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13378c4ccd042e898c389b5e44a3864, State: Initialized, Role: FOLLOWER
I20260924 05:04:42.550851 9650 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13378c4ccd042e898c389b5e44a3864" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35131 } }
I20260924 05:04:42.550927 9650 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:42.550961 9650 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:42.551004 9650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:42.551471 9650 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13378c4ccd042e898c389b5e44a3864" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35131 } }
I20260924 05:04:42.551563 9650 leader_election.cc:302] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13378c4ccd042e898c389b5e44a3864; no voters:
I20260924 05:04:42.551702 9650 leader_election.cc:288] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:42.551748 9655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:42.551927 9650 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:42.551939 9655 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [term 1 LEADER]: Becoming Leader. State: Replica: e13378c4ccd042e898c389b5e44a3864, State: Running, Role: LEADER
I20260924 05:04:42.552119 9655 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13378c4ccd042e898c389b5e44a3864" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35131 } }
I20260924 05:04:42.552423 9655 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13378c4ccd042e898c389b5e44a3864. Latest consensus state: current_term: 1 leader_uuid: "e13378c4ccd042e898c389b5e44a3864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13378c4ccd042e898c389b5e44a3864" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35131 } } }
I20260924 05:04:42.552721 9655 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:42.552656 9657 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13378c4ccd042e898c389b5e44a3864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13378c4ccd042e898c389b5e44a3864" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35131 } } }
I20260924 05:04:42.553131 9657 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:42.553134 9664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:42.553905 9664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:42.554141 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.548032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.0.98.62:35131)
0924 05:04:42.548235 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.548243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.548273 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:04:42.549181 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.549190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.549267 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.549423 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.549603 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.550962 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.551804 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.552448 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.552572 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.553427 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:42.553431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:42.553432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:42.553635 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:42.553667 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:42.553677 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:42.553766 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:42.553865 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 05:04:42.553940 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260924 05:04:42.555383 9664 catalog_manager.cc:1347] Generated new cluster ID: 1180fc6721a94dde945d56fb14c1a0d4
I20260924 05:04:42.555425 9664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:42.575476 9664 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:42.575932 9664 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:42.582221 9664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13378c4ccd042e898c389b5e44a3864: Generated new TSK 0
I20260924 05:04:42.582362 9664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:42.586810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35131
--builtin_ntp_servers=127.0.98.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 05:04:42.665604 9674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:42.665812 9674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:42.665830 9674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:42.666946 9674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:42.666992 9674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:42.667017 9674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:42.668305 9674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:42.668520 9674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:42.668787 9674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:42.674023 9682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:42.674031 9680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.674177 9674 server_base.cc:1034] running on GCE node
W20260924 05:04:42.674039 9679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.674410 9674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:42.674646 9674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:42.675779 9674 hybrid_clock.cc:648] HybridClock initialized: now 1790226282675770 us; error 24 us; skew 500 ppm
I20260924 05:04:42.680182 9674 webserver.cc:460] Webserver started at http://127.0.98.1:42827/ using document root <none> and password file <none>
I20260924 05:04:42.680403 9674 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:42.680450 9674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:42.680584 9674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:42.681594 9674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "d8aa89340b5343efb84ce08f6cdfc93b"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "c40a31390e5df4aa371d4b79bcc9187d"
server_key_iv: "c924a3575499706bec911ed55dab096e"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.681926 9674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d8aa89340b5343efb84ce08f6cdfc93b"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "c40a31390e5df4aa371d4b79bcc9187d"
server_key_iv: "c924a3575499706bec911ed55dab096e"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.683054 9674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:42.683658 9688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.683804 9674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:42.683881 9674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "d8aa89340b5343efb84ce08f6cdfc93b"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "c40a31390e5df4aa371d4b79bcc9187d"
server_key_iv: "c924a3575499706bec911ed55dab096e"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.683945 9674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:42.691866 9674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:42.692128 9674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:42.692245 9674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:42.692437 9674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:42.692735 9674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:42.692768 9674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.692794 9674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:42.692819 9674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.697959 9674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39723
I20260924 05:04:42.698021 9801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39723 every 8 connection(s)
I20260924 05:04:42.698292 9674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:42.701105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9674
I20260924 05:04:42.701179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:42.701350 392 external_mini_cluster.cc:1428] Setting key ee201b132477de801d37615396e33257
I20260924 05:04:42.702179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35131
--builtin_ntp_servers=127.0.98.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 05:04:42.703226 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.698920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58873 (local address 127.0.98.62:35131)
0924 05:04:42.698970 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.698972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.699213 (+ 241us) server_negotiation.cc:407] Connection header received
0924 05:04:42.699711 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.699712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.699727 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.699780 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.700409 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.700901 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.701951 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.702153 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.702208 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.702970 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:42.702971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:42.702972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:42.702976 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:42.702997 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:42.703002 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:42.703047 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:42.703125 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 05:04:42.703149 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:04:42.703424 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.698916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35131 (local address 127.0.98.1:58873)
0924 05:04:42.699100 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:42.699133 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:42.699619 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:42.699844 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:42.699847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:42.699855 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:42.700319 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.700323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.701000 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.701001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.701874 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.701876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.702219 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.702219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.702364 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.702844 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:42.702845 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:42.702871 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:42.702892 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:42.703052 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:42.703057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:42.703060 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:42.703121 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:04:42.703218 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260924 05:04:42.703852 9802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35131
I20260924 05:04:42.703948 9802 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:42.704129 9802 heartbeater.cc:511] Master 127.0.98.62:35131 requested a full tablet report, sending...
I20260924 05:04:42.704600 9614 ts_manager.cc:194] Registered new tserver with Master: d8aa89340b5343efb84ce08f6cdfc93b (127.0.98.1:39723)
I20260924 05:04:42.705176 9614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58873
W20260924 05:04:42.780165 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:42.780356 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:42.780400 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:42.781490 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:42.781529 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:42.781560 9805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:42.782843 9805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:42.783074 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:42.783334 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:42.787758 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:42.787768 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.787961 9805 server_base.cc:1034] running on GCE node
W20260924 05:04:42.788053 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.788255 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:42.788468 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:42.789603 9805 hybrid_clock.cc:648] HybridClock initialized: now 1790226282789592 us; error 28 us; skew 500 ppm
I20260924 05:04:42.790832 9805 webserver.cc:460] Webserver started at http://127.0.98.2:33997/ using document root <none> and password file <none>
I20260924 05:04:42.791074 9805 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:42.791138 9805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:42.791266 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:42.792366 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "09f2c7b1b79a4291aad14ad2e193e47f"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "27ec29a299b3c8747ee389379c2e2021"
server_key_iv: "13b1c4b5eead6a740434ce42bcda9389"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.792691 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "09f2c7b1b79a4291aad14ad2e193e47f"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "27ec29a299b3c8747ee389379c2e2021"
server_key_iv: "13b1c4b5eead6a740434ce42bcda9389"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.793853 9805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:42.794489 9819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.794634 9805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:42.794709 9805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "09f2c7b1b79a4291aad14ad2e193e47f"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "27ec29a299b3c8747ee389379c2e2021"
server_key_iv: "13b1c4b5eead6a740434ce42bcda9389"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.794775 9805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:42.807751 9805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:42.807991 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:42.808106 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:42.808284 9805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:42.808553 9805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:42.808585 9805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.808612 9805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:42.808637 9805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.813679 9805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33017
I20260924 05:04:42.813756 9932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33017 every 8 connection(s)
I20260924 05:04:42.814013 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:42.816694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9805
I20260924 05:04:42.816769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:42.816912 392 external_mini_cluster.cc:1428] Setting key 0dc60388b399e25e54c9a31db6040a0b
I20260924 05:04:42.817819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35131
--builtin_ntp_servers=127.0.98.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 05:04:42.818825 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.814597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41791 (local address 127.0.98.62:35131)
0924 05:04:42.814638 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.814640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.814872 (+ 232us) server_negotiation.cc:407] Connection header received
0924 05:04:42.815334 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.815335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.815343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.815381 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.815956 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.816432 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.817458 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.817758 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.817829 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.818579 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:42.818581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:42.818581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:42.818587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:42.818603 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:42.818608 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:42.818657 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:42.818724 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:04:42.818749 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:42.818995 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.814583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35131 (local address 127.0.98.2:41791)
0924 05:04:42.814753 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:42.814792 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:42.815247 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:42.815433 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:42.815436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:42.815444 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:42.815897 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.815900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.816488 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.816488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.817390 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.817393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.817831 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.817832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.817970 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.818450 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:42.818450 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:42.818477 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:42.818498 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:42.818668 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:42.818671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:42.818673 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:42.818715 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:04:42.818787 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 05:04:42.819419 9933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35131
I20260924 05:04:42.819499 9933 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:42.819664 9933 heartbeater.cc:511] Master 127.0.98.62:35131 requested a full tablet report, sending...
I20260924 05:04:42.819970 9614 ts_manager.cc:194] Registered new tserver with Master: 09f2c7b1b79a4291aad14ad2e193e47f (127.0.98.2:33017)
I20260924 05:04:42.820309 9614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41791
W20260924 05:04:42.894585 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:42.894779 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:42.894806 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:42.895925 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:42.895962 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:42.895993 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:42.897259 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:42.897475 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:42.897742 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:42.902133 9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:42.902144 9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:42.902158 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:42.902280 9936 server_base.cc:1034] running on GCE node
I20260924 05:04:42.902479 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:42.902695 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:42.903813 9936 hybrid_clock.cc:648] HybridClock initialized: now 1790226282903800 us; error 30 us; skew 500 ppm
I20260924 05:04:42.904934 9936 webserver.cc:460] Webserver started at http://127.0.98.3:40543/ using document root <none> and password file <none>
I20260924 05:04:42.905133 9936 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:42.905175 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:42.905293 9936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:42.906292 9936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9d149a3e04040be8feacda1f6f39305"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "bf0d506f3e95d52512aa18a805b4fa7b"
server_key_iv: "7fb2254a48e0bb1228863bbb01edce4a"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.906594 9936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9d149a3e04040be8feacda1f6f39305"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "bf0d506f3e95d52512aa18a805b4fa7b"
server_key_iv: "7fb2254a48e0bb1228863bbb01edce4a"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.907660 9936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:42.908221 9950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.908346 9936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:42.908420 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "d9d149a3e04040be8feacda1f6f39305"
format_stamp: "Formatted at 2026-09-24 05:04:42 on dist-test-slave-m01t"
server_key: "bf0d506f3e95d52512aa18a805b4fa7b"
server_key_iv: "7fb2254a48e0bb1228863bbb01edce4a"
server_key_version: "encryptionkey@0"
I20260924 05:04:42.908475 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:42.918473 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:42.918704 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:42.918812 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:42.918978 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:42.919245 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:42.919281 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.919305 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:42.919329 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:42.924944 9936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34867
I20260924 05:04:42.925602 10063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34867 every 8 connection(s)
I20260924 05:04:42.925812 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:42.932231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 9936
I20260924 05:04:42.932331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:42.932548 392 external_mini_cluster.cc:1428] Setting key 95277a4514bfff0f388032822f9ed051
I20260924 05:04:42.934607 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.928981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54793 (local address 127.0.98.62:35131)
0924 05:04:42.929030 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.929032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.929044 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:04:42.929073 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.929074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.929086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.929145 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.930172 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.930907 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.932928 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.933190 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.933233 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.934347 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:42.934349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:42.934349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:42.934355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:42.934373 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:42.934378 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:42.934433 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:42.934522 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 05:04:42.934546 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:04:42.934958 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:42.926524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35131 (local address 127.0.98.3:54793)
0924 05:04:42.926771 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:42.926812 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:42.927336 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:42.929284 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:42.929289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:42.929301 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:42.930031 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.930039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.931009 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.931010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.932835 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:42.932839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:42.933403 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:42.933404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:42.933572 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:42.934181 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:42.934182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:42.934212 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:42.934237 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:42.934438 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:42.934444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:42.934447 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:42.934526 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 05:04:42.934732 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":2426,"threads_started":1}
I20260924 05:04:42.935415 10064 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35131
I20260924 05:04:42.935509 10064 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:42.935683 10064 heartbeater.cc:511] Master 127.0.98.62:35131 requested a full tablet report, sending...
I20260924 05:04:42.936043 9614 ts_manager.cc:194] Registered new tserver with Master: d9d149a3e04040be8feacda1f6f39305 (127.0.98.3:34867)
I20260924 05:04:42.936499 9614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54793
I20260924 05:04:42.944634 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 05:04:42.946365 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:42.945846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.98.62:35131)
0924 05:04:42.945883 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:42.945885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:42.946012 (+ 127us) server_negotiation.cc:407] Connection header received
0924 05:04:42.946108 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:42.946109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:42.946118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:42.946164 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:42.946301 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43312: BlockingRecv error: recv got EOF from 127.0.0.1:43312 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:42.946772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9674
I20260924 05:04:42.952443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9805
I20260924 05:04:42.958521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9936
I20260924 05:04:42.963088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 9583
2026-09-24T05:04:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T05:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:42Z Disabled control of system clock
I20260924 05:04:42.987211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45865
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:04:43.063865 10075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:43.064061 10075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:43.064080 10075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:43.065199 10075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:43.065240 10075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:43.065251 10075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:43.065264 10075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:43.065286 10075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:43.066594 10075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:43.066811 10075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:43.067061 10075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:43.071431 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:43.071488 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:43.071507 10075 server_base.cc:1034] running on GCE node
W20260924 05:04:43.071427 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:43.071839 10075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:43.072093 10075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:43.073218 10075 hybrid_clock.cc:648] HybridClock initialized: now 1790226283073209 us; error 26 us; skew 500 ppm
I20260924 05:04:43.074311 10075 webserver.cc:460] Webserver started at http://127.0.98.62:33411/ using document root <none> and password file <none>
I20260924 05:04:43.074502 10075 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:43.074545 10075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:43.074654 10075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:43.075632 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "bcbcfb14e00744df9ec9d71bb20f1003"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "c887a6882e4ba5d27d423e69cf94e1c6"
server_key_iv: "b620e50661385d0e4e912f4d2f23eb73"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.075937 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "bcbcfb14e00744df9ec9d71bb20f1003"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "c887a6882e4ba5d27d423e69cf94e1c6"
server_key_iv: "b620e50661385d0e4e912f4d2f23eb73"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.077060 10075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:43.077641 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.077790 10075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:43.077865 10075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "bcbcfb14e00744df9ec9d71bb20f1003"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "c887a6882e4ba5d27d423e69cf94e1c6"
server_key_iv: "b620e50661385d0e4e912f4d2f23eb73"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.077921 10075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:43.081111 10075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:43.081336 10075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:43.081449 10075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:43.084851 10075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45865
I20260924 05:04:43.084878 10141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45865 every 8 connection(s)
I20260924 05:04:43.085155 10075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:43.085705 10142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:43.088176 10142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003: Bootstrap starting.
I20260924 05:04:43.088687 10142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:43.088913 10142 log.cc:824] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:43.089488 10142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003: No bootstrap required, opened a new log
I20260924 05:04:43.090471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10075
I20260924 05:04:43.090551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:43.090705 392 external_mini_cluster.cc:1428] Setting key e2ad8ca204618ff857681443e5becbec
I20260924 05:04:43.091159 10142 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260924 05:04:43.091331 10142 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:43.091359 10142 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbcfb14e00744df9ec9d71bb20f1003, State: Initialized, Role: FOLLOWER
I20260924 05:04:43.091490 10142 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260924 05:04:43.091553 10142 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:43.091574 10142 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:43.091619 10142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:43.092103 10142 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260924 05:04:43.092195 10142 leader_election.cc:302] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcbcfb14e00744df9ec9d71bb20f1003; no voters:
I20260924 05:04:43.092347 10142 leader_election.cc:288] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:43.092383 10147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:43.092487 10147 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [term 1 LEADER]: Becoming Leader. State: Replica: bcbcfb14e00744df9ec9d71bb20f1003, State: Running, Role: LEADER
I20260924 05:04:43.092571 10147 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260924 05:04:43.092597 10142 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:43.092934 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcbcfb14e00744df9ec9d71bb20f1003. Latest consensus state: current_term: 1 leader_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } } }
I20260924 05:04:43.093017 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:43.093163 10149 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbcfb14e00744df9ec9d71bb20f1003" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } } }
I20260924 05:04:43.093220 10149 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:43.093325 10153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:43.093478 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.091169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.62:45865)
0924 05:04:43.091361 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:43.091370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:43.091402 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:04:43.091518 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:43.091524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:43.091566 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:43.091700 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:43.092617 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:43.092621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:43.092622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:43.092626 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:43.092670 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:43.092684 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:43.092732 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:43.092868 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 05:04:43.092929 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 05:04:43.094098 10153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:43.098779 10153 catalog_manager.cc:1347] Generated new cluster ID: c07cd3e248a5480fa885eb4a36373e9b
I20260924 05:04:43.098835 10153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:04:43.110065 10153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:43.110220 10153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:43.113994 10153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcbcfb14e00744df9ec9d71bb20f1003: Generated new TSK 0
I20260924 05:04:43.114156 10153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:04:43.120401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:04:43.199290 10166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:43.199482 10166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:43.199503 10166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:43.200656 10166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:43.200692 10166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:43.200717 10166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:43.202355 10166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:43.202574 10166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:43.202836 10166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:43.208124 10172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:43.208216 10174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:43.208158 10171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:43.208616 10166 server_base.cc:1034] running on GCE node
I20260924 05:04:43.208802 10166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:43.209024 10166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:43.210160 10166 hybrid_clock.cc:648] HybridClock initialized: now 1790226283210145 us; error 31 us; skew 500 ppm
I20260924 05:04:43.211323 10166 webserver.cc:460] Webserver started at http://127.0.98.1:42759/ using document root <none> and password file <none>
I20260924 05:04:43.211521 10166 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:43.211566 10166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:43.211683 10166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:43.212666 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "c07051ee465f477b991c34e887708984"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "e352a7d98220267c712711156f259f94"
server_key_iv: "f0216da77601fba9de8ed45d3b6db7c4"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.212981 10166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c07051ee465f477b991c34e887708984"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "e352a7d98220267c712711156f259f94"
server_key_iv: "f0216da77601fba9de8ed45d3b6db7c4"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.214214 10166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:04:43.214906 10180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.215049 10166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:43.215126 10166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "c07051ee465f477b991c34e887708984"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "e352a7d98220267c712711156f259f94"
server_key_iv: "f0216da77601fba9de8ed45d3b6db7c4"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.215189 10166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:43.218262 10166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:43.218493 10166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:43.218602 10166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:43.218787 10166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:43.219050 10166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:43.219080 10166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.219108 10166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:43.219131 10166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.224385 10166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45841
I20260924 05:04:43.224458 10293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45841 every 8 connection(s)
I20260924 05:04:43.224830 10166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:04:43.232342 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.225482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58027 (local address 127.0.98.62:45865)
0924 05:04:43.225552 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:43.225555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:43.225801 (+ 246us) server_negotiation.cc:407] Connection header received
0924 05:04:43.226302 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:43.226303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:43.226309 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:43.226358 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:43.232118 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:43.232119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:43.232120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:43.232125 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:43.232145 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:43.232150 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:43.232202 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:43.232264 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 05:04:43.232279 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:04:43.232563 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.225464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.1:58027)
0924 05:04:43.225669 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:43.225702 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:43.226217 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:43.226426 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:43.226429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:43.226437 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:43.231921 (+ 5484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:43.231922 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:43.231962 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:43.231998 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:43.232206 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:43.232210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:43.232213 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:43.232270 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 05:04:43.232329 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260924 05:04:43.233047 10294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260924 05:04:43.233137 10294 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:43.233284 10294 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260924 05:04:43.233811 10106 ts_manager.cc:194] Registered new tserver with Master: c07051ee465f477b991c34e887708984 (127.0.98.1:45841)
I20260924 05:04:43.234740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10166
I20260924 05:04:43.234815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:43.234958 392 external_mini_cluster.cc:1428] Setting key c9788df3a80a0c565b0d3b3f450fb5be
I20260924 05:04:43.235775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:04:43.313191 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:43.313365 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:43.313385 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:43.314457 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:43.314491 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:43.314513 10297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:43.315738 10297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:43.315905 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:43.316133 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:43.320637 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:43.320688 10297 server_base.cc:1034] running on GCE node
W20260924 05:04:43.320626 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:43.320642 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:43.320916 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:43.321123 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:43.322248 10297 hybrid_clock.cc:648] HybridClock initialized: now 1790226283322237 us; error 28 us; skew 500 ppm
I20260924 05:04:43.323316 10297 webserver.cc:460] Webserver started at http://127.0.98.2:33293/ using document root <none> and password file <none>
I20260924 05:04:43.323524 10297 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:43.323566 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:43.323683 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:43.324654 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f4e18d8d63f4c4f972ad70de11e2ee5"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "b1935edb2d96c6083ca68a8255cfb46f"
server_key_iv: "2e508f159d8a27f22ad79b888f8a1ecf"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.324951 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f4e18d8d63f4c4f972ad70de11e2ee5"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "b1935edb2d96c6083ca68a8255cfb46f"
server_key_iv: "2e508f159d8a27f22ad79b888f8a1ecf"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.326110 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:04:43.326712 10311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.326853 10297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:04:43.326923 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "2f4e18d8d63f4c4f972ad70de11e2ee5"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "b1935edb2d96c6083ca68a8255cfb46f"
server_key_iv: "2e508f159d8a27f22ad79b888f8a1ecf"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.326985 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:43.330147 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:43.330371 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:43.330480 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:43.330650 10297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:43.330922 10297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:43.330952 10297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.330979 10297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:43.331002 10297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.336288 10297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46315
I20260924 05:04:43.336347 10424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46315 every 8 connection(s)
I20260924 05:04:43.336623 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:04:43.339432 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.337282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60857 (local address 127.0.98.62:45865)
0924 05:04:43.337331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:43.337334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:43.337562 (+ 228us) server_negotiation.cc:407] Connection header received
0924 05:04:43.338097 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:43.338099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:43.338105 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:43.338169 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:43.339255 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:43.339257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:43.339257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:43.339261 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:43.339278 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:43.339282 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:43.339328 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:43.339370 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 05:04:43.339382 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:04:43.339609 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.337268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.2:60857)
0924 05:04:43.337439 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:43.337473 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:43.337980 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:43.338287 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:43.338291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:43.338299 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:43.339122 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:43.339122 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:43.339150 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:43.339177 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:43.339324 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:43.339327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:43.339329 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:43.339376 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:04:43.339421 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260924 05:04:43.339859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10297
I20260924 05:04:43.339939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:43.340044 10425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260924 05:04:43.340127 10425 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:43.340134 392 external_mini_cluster.cc:1428] Setting key 9bb974f107bcec22168ca0a87fe59e45
I20260924 05:04:43.340262 10425 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260924 05:04:43.340596 10105 ts_manager.cc:194] Registered new tserver with Master: 2f4e18d8d63f4c4f972ad70de11e2ee5 (127.0.98.2:46315)
I20260924 05:04:43.341007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--builtin_ntp_servers=127.0.98.20:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:04:43.416497 10428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:43.416675 10428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:43.416694 10428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:43.417860 10428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:43.417898 10428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:43.417925 10428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:43.419236 10428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:43.419442 10428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:43.419689 10428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:43.424189 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:43.424171 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:43.424160 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:43.424204 10428 server_base.cc:1034] running on GCE node
I20260924 05:04:43.424502 10428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:43.424707 10428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:43.425834 10428 hybrid_clock.cc:648] HybridClock initialized: now 1790226283425821 us; error 29 us; skew 500 ppm
I20260924 05:04:43.426952 10428 webserver.cc:460] Webserver started at http://127.0.98.3:37645/ using document root <none> and password file <none>
I20260924 05:04:43.427161 10428 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:43.427205 10428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:43.427318 10428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:43.428349 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "6292050e41bf4abfad8e95ea0bf76921"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "6fd78f4bfd432050b54911c418b49ba7"
server_key_iv: "960567aefa5af459f26c9f07a9bca5a9"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.428659 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6292050e41bf4abfad8e95ea0bf76921"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "6fd78f4bfd432050b54911c418b49ba7"
server_key_iv: "960567aefa5af459f26c9f07a9bca5a9"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.429931 10428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:43.430610 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.430742 10428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:43.430817 10428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "6292050e41bf4abfad8e95ea0bf76921"
format_stamp: "Formatted at 2026-09-24 05:04:43 on dist-test-slave-m01t"
server_key: "6fd78f4bfd432050b54911c418b49ba7"
server_key_iv: "960567aefa5af459f26c9f07a9bca5a9"
server_key_version: "encryptionkey@0"
I20260924 05:04:43.430879 10428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:43.434062 10428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:43.434300 10428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:43.434424 10428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:43.434609 10428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:43.434873 10428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:43.434904 10428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.434932 10428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:43.434954 10428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:43.440111 10428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46795
I20260924 05:04:43.440168 10555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46795 every 8 connection(s)
I20260924 05:04:43.440450 10428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:43.442994 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.441066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35011 (local address 127.0.98.62:45865)
0924 05:04:43.441123 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:43.441125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:43.441328 (+ 203us) server_negotiation.cc:407] Connection header received
0924 05:04:43.441789 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:43.441790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:43.441796 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:43.441837 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:43.442813 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:43.442814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:43.442814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:04:43.442817 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:43.442830 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:04:43.442833 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:43.442875 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:43.442930 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 05:04:43.442941 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:04:43.443150 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:43.441066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.3:35011)
0924 05:04:43.441218 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:43.441249 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:43.441686 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:43.441888 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:43.441891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:43.441899 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:43.442695 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:04:43.442696 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:43.442723 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:04:43.442748 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:43.442878 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:43.442881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:43.442883 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:43.442932 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:04:43.442978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":79,"threads_started":1}
I20260924 05:04:43.443576 10556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260924 05:04:43.443655 10556 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:43.443778 10556 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260924 05:04:43.444056 10105 ts_manager.cc:194] Registered new tserver with Master: 6292050e41bf4abfad8e95ea0bf76921 (127.0.98.3:46795)
I20260924 05:04:43.444623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10428
I20260924 05:04:43.444695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:43.444821 392 external_mini_cluster.cc:1428] Setting key 45fda561d7690a7a9f633bee329eb18d
I20260924 05:04:43.445531 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 05:04:43.447103 10146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:04:43.446680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.0.98.62:45865)
0924 05:04:43.446731 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:43.446732 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:43.446829 (+ 97us) server_negotiation.cc:407] Connection header received
0924 05:04:43.446888 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:43.446889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:43.446892 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:43.446928 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:43.447051 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38322: BlockingRecv error: recv got EOF from 127.0.0.1:38322 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:04:43.447144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10166
I20260924 05:04:43.451490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10297
I20260924 05:04:43.456079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10428
I20260924 05:04:43.459765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10075
2026-09-24T05:04:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (496 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:43 dist-test-slave-m01t krb5kdc[10566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:43 dist-test-slave-m01t krb5kdc[10566](info): set up 2 sockets
Sep 24 05:04:43 dist-test-slave-m01t krb5kdc[10566](info): commencing operation
krb5kdc: starting...
W20260924 05:04:44.406116 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:04:44.509896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:44.585592 10579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:44.585776 10579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:44.585798 10579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:44.586843 10579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:04:44.586874 10579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:44.586885 10579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:44.586896 10579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:04:44.586913 10579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:04:44.588220 10579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:44.588388 10579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:44.588541 10579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:04:44.592978 10579 server_base.cc:1034] running on GCE node
W20260924 05:04:44.592998 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:44.593096 10584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:44.592978 10585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:44.593345 10579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:44.593567 10579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:44.594694 10579 hybrid_clock.cc:648] HybridClock initialized: now 1790226284594678 us; error 34 us; skew 500 ppm
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:44.596868 10579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:04:44.597258 10579 webserver.cc:460] Webserver started at http://127.0.98.62:42779/ using document root <none> and password file <none>
I20260924 05:04:44.597460 10579 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:44.597501 10579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:44.597608 10579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:44.598587 10579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "21fe408b756d428892d22f50c2cd2656"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "0534b01e04512adc5280aed27bc74c3b"
server_key_iv: "878c8a653b774f633c6b5d639b7efac6"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.598877 10579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "21fe408b756d428892d22f50c2cd2656"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "0534b01e04512adc5280aed27bc74c3b"
server_key_iv: "878c8a653b774f633c6b5d639b7efac6"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.599998 10579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:04:44.600579 10594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.600739 10579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:44.600816 10579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "21fe408b756d428892d22f50c2cd2656"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "0534b01e04512adc5280aed27bc74c3b"
server_key_iv: "878c8a653b774f633c6b5d639b7efac6"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.600865 10579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:44.614454 10579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:44.615936 10579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:44.616031 10579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:44.619383 10579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40091
I20260924 05:04:44.619423 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40091 every 8 connection(s)
I20260924 05:04:44.619694 10579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:04:44.620561 10647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:44.622633 10647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656: Bootstrap starting.
I20260924 05:04:44.623140 10647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:44.623344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10579
I20260924 05:04:44.623382 10647 log.cc:824] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:44.623420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:04:44.623587 392 external_mini_cluster.cc:1428] Setting key 2f1e9a342e7b00f678aa84f851ed6611
I20260924 05:04:44.624037 10647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656: No bootstrap required, opened a new log
I20260924 05:04:44.626012 10647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fe408b756d428892d22f50c2cd2656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40091 } }
I20260924 05:04:44.626145 10647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:44.626166 10647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe408b756d428892d22f50c2cd2656, State: Initialized, Role: FOLLOWER
I20260924 05:04:44.626269 10647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fe408b756d428892d22f50c2cd2656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40091 } }
I20260924 05:04:44.626336 10647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:04:44.626360 10647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:04:44.626405 10647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:44.626909 10647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fe408b756d428892d22f50c2cd2656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40091 } }
I20260924 05:04:44.626993 10647 leader_election.cc:302] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21fe408b756d428892d22f50c2cd2656; no voters:
I20260924 05:04:44.627140 10647 leader_election.cc:288] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:04:44.627172 10652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:44.627310 10652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [term 1 LEADER]: Becoming Leader. State: Replica: 21fe408b756d428892d22f50c2cd2656, State: Running, Role: LEADER
I20260924 05:04:44.627383 10647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:04:44.627422 10652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fe408b756d428892d22f50c2cd2656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40091 } }
I20260924 05:04:44.627719 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21fe408b756d428892d22f50c2cd2656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fe408b756d428892d22f50c2cd2656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40091 } } }
I20260924 05:04:44.627872 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:44.628041 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21fe408b756d428892d22f50c2cd2656. Latest consensus state: current_term: 1 leader_uuid: "21fe408b756d428892d22f50c2cd2656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21fe408b756d428892d22f50c2cd2656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40091 } } }
I20260924 05:04:44.628119 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656 [sys.catalog]: This master's current role is: LEADER
I20260924 05:04:44.628432 10661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:04:44.628928 10661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:04:44.630646 10661 catalog_manager.cc:1347] Generated new cluster ID: 5028ebba54784dd0a7daa40bc9cc44be
I20260924 05:04:44.630694 10661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:44.635177 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:44.624012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.98.62:40091)
0924 05:04:44.624196 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:44.624204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:44.624228 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:04:44.625113 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:44.625120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:44.625167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:44.625274 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:44.625993 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.627219 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.628220 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.628817 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.628917 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:44.631522 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:44.631525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:44.631526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:44.631538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:44.633425 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:44.633792 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:44.633795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:44.633796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:44.633889 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:44.634053 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:44.634053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:44.634054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:44.634328 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:44.634452 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:44.634849 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 05:04:44.634974 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260924 05:04:44.645433 10661 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:04:44.645977 10661 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:04:44.656037 10661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656: Generated new TSK 0
I20260924 05:04:44.656176 10661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:04:44.684260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40091
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:44.761922 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:44.762112 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:44.762140 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:44.763248 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:44.763288 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:44.763312 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:04:44.764694 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:44.764901 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:44.765103 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:44.769500 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:44.769521 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:44.769578 10673 server_base.cc:1034] running on GCE node
W20260924 05:04:44.769502 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:44.769929 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:44.770145 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:44.771271 10673 hybrid_clock.cc:648] HybridClock initialized: now 1790226284771263 us; error 27 us; skew 500 ppm
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:44.773545 10673 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:04:44.773947 10673 webserver.cc:460] Webserver started at http://127.0.98.1:33949/ using document root <none> and password file <none>
I20260924 05:04:44.774151 10673 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:44.774207 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:44.774307 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:44.775220 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "9474d5b9847f4a47908265e5e392efcb"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "88e4b0353f324bcb4cff8b15163a5db6"
server_key_iv: "6dd0327e421b131fb25c9811b8b187f5"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.775491 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9474d5b9847f4a47908265e5e392efcb"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "88e4b0353f324bcb4cff8b15163a5db6"
server_key_iv: "6dd0327e421b131fb25c9811b8b187f5"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.776589 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 05:04:44.777171 10688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.777302 10673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:44.777370 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "9474d5b9847f4a47908265e5e392efcb"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "88e4b0353f324bcb4cff8b15163a5db6"
server_key_iv: "6dd0327e421b131fb25c9811b8b187f5"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.777431 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:44.784366 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:44.785920 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:44.786034 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:44.786207 10673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:44.786451 10673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:44.786482 10673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.786501 10673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:44.786525 10673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.791733 10673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46291
I20260924 05:04:44.791790 10801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46291 every 8 connection(s)
I20260924 05:04:44.792047 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:44.798032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10673
I20260924 05:04:44.798143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:04:44.798314 392 external_mini_cluster.cc:1428] Setting key a2ce9a1f151861e166d5a13f3c10779c
I20260924 05:04:44.800666 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:44.792663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36947 (local address 127.0.98.62:40091)
0924 05:04:44.792715 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:44.792717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:44.792879 (+ 162us) server_negotiation.cc:407] Connection header received
0924 05:04:44.793358 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:44.793360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:44.793372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:44.793421 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:44.794361 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.794873 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.795829 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.796043 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.796098 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:44.798101 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:44.798102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:44.798102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:44.798111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:44.799367 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:44.799783 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:44.799784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:44.799784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:44.799829 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:44.800069 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:44.800070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:44.800070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:44.800196 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:44.800259 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:44.800480 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 05:04:44.800574 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:04:44.800716 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:44.792638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40091 (local address 127.0.98.1:36947)
0924 05:04:44.792765 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:44.792805 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:44.793270 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:44.793489 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:44.793493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:44.794084 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:44.794269 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:44.794274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.794953 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.794954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:44.795747 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:44.795750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.796110 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.796111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:44.796222 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:44.796668 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:44.796676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:44.797997 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:44.799440 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:44.799446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:44.799448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:44.799720 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:44.799880 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:44.799881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:44.799881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:44.800009 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:44.800252 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:44.800256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:44.800373 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 05:04:44.800450 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:04:44.800560 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260924 05:04:44.801470 10802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40091
I20260924 05:04:44.801568 10802 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:44.801760 10802 heartbeater.cc:511] Master 127.0.98.62:40091 requested a full tablet report, sending...
I20260924 05:04:44.802240 10611 ts_manager.cc:194] Registered new tserver with Master: 9474d5b9847f4a47908265e5e392efcb (127.0.98.1:46291)
I20260924 05:04:44.802807 10611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36947
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:04:44.820415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40091
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:44.897126 10807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:44.897320 10807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:44.897349 10807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:44.898458 10807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:44.898497 10807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:44.898530 10807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:04:44.899897 10807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:44.900128 10807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:44.900321 10807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:44.904662 10813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:44.904687 10815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:44.904701 10812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:44.904827 10807 server_base.cc:1034] running on GCE node
I20260924 05:04:44.905037 10807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:44.905234 10807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:44.906360 10807 hybrid_clock.cc:648] HybridClock initialized: now 1790226284906338 us; error 38 us; skew 500 ppm
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:44.908485 10807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:04:44.908828 10807 webserver.cc:460] Webserver started at http://127.0.98.2:42665/ using document root <none> and password file <none>
I20260924 05:04:44.909001 10807 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:44.909039 10807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:44.909121 10807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:44.910064 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "97e9f31a4b6e4e4090e8479c5715e6ac"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "11ef014cc612f8370fd5d7e1180839a3"
server_key_iv: "c539eef2a4d2f920c49d4393afbe2e7c"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.910338 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97e9f31a4b6e4e4090e8479c5715e6ac"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "11ef014cc612f8370fd5d7e1180839a3"
server_key_iv: "c539eef2a4d2f920c49d4393afbe2e7c"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.911398 10807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:04:44.911954 10822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.912092 10807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:44.912155 10807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "97e9f31a4b6e4e4090e8479c5715e6ac"
format_stamp: "Formatted at 2026-09-24 05:04:44 on dist-test-slave-m01t"
server_key: "11ef014cc612f8370fd5d7e1180839a3"
server_key_iv: "c539eef2a4d2f920c49d4393afbe2e7c"
server_key_version: "encryptionkey@0"
I20260924 05:04:44.912221 10807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:44.919925 10807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:44.921450 10807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:44.921563 10807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:44.921785 10807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:44.922055 10807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:44.922087 10807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.922106 10807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:44.922130 10807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:44.927537 10807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260924 05:04:44.927573 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260924 05:04:44.927871 10807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:04:44 dist-test-slave-m01t krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:44.934001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10807
I20260924 05:04:44.934109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:04:44.934295 392 external_mini_cluster.cc:1428] Setting key 3bc52b66ec38d21d25fffdcb32221389
I20260924 05:04:44.936731 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:44.928582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39641 (local address 127.0.98.62:40091)
0924 05:04:44.928626 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:44.928628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:44.928818 (+ 190us) server_negotiation.cc:407] Connection header received
0924 05:04:44.929316 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:44.929317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:44.929337 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:44.929384 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:44.930332 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.930824 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.931747 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.931926 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.931967 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:44.933946 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:44.933948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:44.933948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:44.933958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:44.935437 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:44.935816 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:44.935817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:44.935817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:44.935877 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:44.936075 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:44.936076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:44.936076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:44.936208 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:44.936277 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:44.936600 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 05:04:44.936665 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:04:44.936992 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:44.928569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40091 (local address 127.0.98.2:39641)
0924 05:04:44.928702 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:44.928738 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:44.929235 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:44.929440 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:44.929443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:44.930038 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:44.930254 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:44.930260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.930890 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.930890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:44.931683 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:44.931685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:44.931979 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:44.931980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:44.932083 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:44.932509 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:44.932517 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:44.933847 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:44.935526 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:44.935532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:44.935534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:44.935756 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:44.935931 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:44.935932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:44.935932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:44.936027 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:44.936285 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:44.936292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:44.936464 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 05:04:44.936587 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 05:04:44.936759 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260924 05:04:44.937501 10936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40091
I20260924 05:04:44.937599 10936 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:44.937809 10936 heartbeater.cc:511] Master 127.0.98.62:40091 requested a full tablet report, sending...
I20260924 05:04:44.938184 10611 ts_manager.cc:194] Registered new tserver with Master: 97e9f31a4b6e4e4090e8479c5715e6ac (127.0.98.2:45319)
I20260924 05:04:44.938674 10611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39641
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:04:44.956174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40091
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:04:45.033557 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:04:45.033794 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:04:45.033826 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:04:45.035012 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:04:45.035059 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:04:45.035094 10941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:04:45.036545 10941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:04:45.036794 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:04:45.037005 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:04:45.041695 10949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:45.041749 10941 server_base.cc:1034] running on GCE node
W20260924 05:04:45.041751 10946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:04:45.041821 10947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:04:45.042191 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:04:45.042430 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:04:45.043563 10941 hybrid_clock.cc:648] HybridClock initialized: now 1790226285043548 us; error 31 us; skew 500 ppm
Sep 24 05:04:45 dist-test-slave-m01t krb5kdc[10566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:04:45.045864 10941 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:04:45.046247 10941 webserver.cc:460] Webserver started at http://127.0.98.3:44739/ using document root <none> and password file <none>
I20260924 05:04:45.046439 10941 fs_manager.cc:359] Metadata directory not provided
I20260924 05:04:45.046496 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:04:45.046620 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:04:45.047602 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d13424f23af4bb5ac18faa43e4c5e46"
format_stamp: "Formatted at 2026-09-24 05:04:45 on dist-test-slave-m01t"
server_key: "4890bfddd0379d7bb3c9b95946b5afca"
server_key_iv: "73f2d595497d45322b8c3454e52bd616"
server_key_version: "encryptionkey@0"
I20260924 05:04:45.047932 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d13424f23af4bb5ac18faa43e4c5e46"
format_stamp: "Formatted at 2026-09-24 05:04:45 on dist-test-slave-m01t"
server_key: "4890bfddd0379d7bb3c9b95946b5afca"
server_key_iv: "73f2d595497d45322b8c3454e52bd616"
server_key_version: "encryptionkey@0"
I20260924 05:04:45.049106 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:04:45.049746 10956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:45.049899 10941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:04:45.049975 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "1d13424f23af4bb5ac18faa43e4c5e46"
format_stamp: "Formatted at 2026-09-24 05:04:45 on dist-test-slave-m01t"
server_key: "4890bfddd0379d7bb3c9b95946b5afca"
server_key_iv: "73f2d595497d45322b8c3454e52bd616"
server_key_version: "encryptionkey@0"
I20260924 05:04:45.050040 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:04:45.058581 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:04:45.060470 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:04:45.060600 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:04:45.060789 10941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:04:45.061041 10941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:04:45.061071 10941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:45.061096 10941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:04:45.061120 10941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:04:45.066257 10941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41027
I20260924 05:04:45.066313 11069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41027 every 8 connection(s)
I20260924 05:04:45.066555 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:04:45.069697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 10941
I20260924 05:04:45.069813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:04:45.069988 392 external_mini_cluster.cc:1428] Setting key 62ba95f7fa1db75199e393736c9f85e0
Sep 24 05:04:45 dist-test-slave-m01t krb5kdc[10566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:04:45.074510 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.067102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60641 (local address 127.0.98.62:40091)
0924 05:04:45.067145 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.067148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.067325 (+ 177us) server_negotiation.cc:407] Connection header received
0924 05:04:45.067752 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.067753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.067763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.067807 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:45.068624 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.069104 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.070093 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.070362 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.070416 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.072211 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:45.072212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:45.072212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:45.072219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:45.073288 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:45.073632 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:45.073633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:45.073634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:45.073690 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:45.073922 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:45.073922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:45.073923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:45.074044 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:45.074100 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.074342 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.074420 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:45.074669 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.067092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40091 (local address 127.0.98.3:60641)
0924 05:04:45.067227 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.067257 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:45.067685 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:45.067856 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:45.067859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:45.068375 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:04:45.068559 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.068564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.069162 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.069163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.070017 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.070019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.070429 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.070430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.070559 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.070955 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:04:45.070962 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:04:45.072131 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:04:45.073358 (+ 1227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:45.073364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:45.073366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:45.073569 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:45.073767 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:04:45.073767 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:04:45.073768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:04:45.073868 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:04:45.074098 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:04:45.074104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:04:45.074231 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 05:04:45.074316 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 05:04:45.074455 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260924 05:04:45.075023 11070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40091
I20260924 05:04:45.075120 11070 heartbeater.cc:464] Registering TS with master...
I20260924 05:04:45.075273 11070 heartbeater.cc:511] Master 127.0.98.62:40091 requested a full tablet report, sending...
I20260924 05:04:45.075556 10611 ts_manager.cc:194] Registered new tserver with Master: 1d13424f23af4bb5ac18faa43e4c5e46 (127.0.98.3:41027)
I20260924 05:04:45.075879 10611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60641
I20260924 05:04:45.081341 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:04:45.087656 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.082525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.98.62:40091)
0924 05:04:45.082582 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.082584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.082688 (+ 104us) server_negotiation.cc:407] Connection header received
0924 05:04:45.082736 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.082736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.082745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.082784 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:04:45.083291 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.083746 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.084421 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.084616 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.084666 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.085671 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:04:45.085672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:04:45.085672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:04:45.085683 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:04:45.086710 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:45.086981 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:45.086982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:45.086982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:45.087030 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:04:45.087186 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:04:45.087186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:04:45.087187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:04:45.087315 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:04:45.087398 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.087501 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.087564 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:04:45.088918 10611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:04:45.090495 10611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:04:45.096312 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.092515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:59634)
0924 05:04:45.092779 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.092789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:45.092830 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:45.093410 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:45.093411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:45.093413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:45.093449 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.093450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.094575 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.094576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.095428 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.095430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.096044 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.096045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.096117 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.096129 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 05:04:45.096192 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:04:45.096240 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260924 05:04:45.096321 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.092600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.0.98.2:45319)
0924 05:04:45.093016 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.093019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.093028 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:04:45.093062 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.093064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.093087 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.093308 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:45.093525 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.094488 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.095501 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.095967 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.096030 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.096086 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.096210 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.096245 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":87,"threads_started":1}
I20260924 05:04:45.096623 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.092422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.0.98.3:41027)
0924 05:04:45.092652 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.092657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.092668 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:04:45.092705 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.092707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.092722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.092807 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:45.093023 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.094126 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.095528 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.096241 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.096304 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.096354 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.096510 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.096551 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260924 05:04:45.096558 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.092679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46291 (local address 127.0.0.1:59606)
0924 05:04:45.092792 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.092801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:45.092844 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:45.093105 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:45.093106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:45.093108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:45.093137 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.093138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.094111 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.094112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.095434 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.095435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.096335 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.096336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.096394 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.096398 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:45.096425 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 05:04:45.096463 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":32,"thread_start_us":63,"threads_started":1}
I20260924 05:04:45.096827 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.092807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.98.1:46291)
0924 05:04:45.092959 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.092963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.092973 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:04:45.093009 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.093011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.093027 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.093105 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:45.093180 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.094029 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.095528 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.096240 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.096620 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.096685 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.096713 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.096756 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260924 05:04:45.096611 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.092338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41027 (local address 127.0.0.1:50144)
0924 05:04:45.092443 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.092482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:45.092538 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:45.092891 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:45.092894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:45.092896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:45.092952 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.092953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.094217 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.094218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.095432 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.095434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.096381 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.096382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.096431 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.096434 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:04:45.096478 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:04:45.096510 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":3,"thread_start_us":45,"threads_started":1}
I20260924 05:04:45.097398 11004 tablet_service.cc:1467] Processing CreateTablet for tablet c5815041674a4e028f2b6d91f72f404e (DEFAULT_TABLE table=test-table [id=a8563ff4d4d04dfc9759e46ab0f0b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:45.097415 10870 tablet_service.cc:1467] Processing CreateTablet for tablet c5815041674a4e028f2b6d91f72f404e (DEFAULT_TABLE table=test-table [id=a8563ff4d4d04dfc9759e46ab0f0b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:45.097424 10736 tablet_service.cc:1467] Processing CreateTablet for tablet c5815041674a4e028f2b6d91f72f404e (DEFAULT_TABLE table=test-table [id=a8563ff4d4d04dfc9759e46ab0f0b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:04:45.097707 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5815041674a4e028f2b6d91f72f404e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:45.097733 10736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5815041674a4e028f2b6d91f72f404e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:45.097733 10870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5815041674a4e028f2b6d91f72f404e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:04:45.100085 11084 tablet_bootstrap.cc:492] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Bootstrap starting.
I20260924 05:04:45.100625 11084 tablet_bootstrap.cc:654] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:45.100826 11085 tablet_bootstrap.cc:492] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb: Bootstrap starting.
I20260924 05:04:45.100951 11086 tablet_bootstrap.cc:492] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Bootstrap starting.
I20260924 05:04:45.101248 11084 log.cc:824] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:45.101480 11085 tablet_bootstrap.cc:654] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:45.101523 11086 tablet_bootstrap.cc:654] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Neither blocks nor log segments found. Creating new log.
I20260924 05:04:45.102020 11085 log.cc:824] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:45.102072 11086 log.cc:824] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Log is configured to *not* fsync() on all Append() calls
I20260924 05:04:45.102154 11084 tablet_bootstrap.cc:492] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: No bootstrap required, opened a new log
I20260924 05:04:45.102237 11084 ts_tablet_manager.cc:1397] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:45.102664 11085 tablet_bootstrap.cc:492] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb: No bootstrap required, opened a new log
I20260924 05:04:45.102730 11085 ts_tablet_manager.cc:1397] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:45.102730 11086 tablet_bootstrap.cc:492] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: No bootstrap required, opened a new log
I20260924 05:04:45.102792 11086 ts_tablet_manager.cc:1397] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:04:45.104166 11084 raft_consensus.cc:348] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.104329 11084 raft_consensus.cc:374] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:45.104362 11084 raft_consensus.cc:729] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d13424f23af4bb5ac18faa43e4c5e46, State: Initialized, Role: FOLLOWER
I20260924 05:04:45.104317 11086 raft_consensus.cc:348] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.104430 11086 raft_consensus.cc:374] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:45.104457 11086 raft_consensus.cc:729] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e9f31a4b6e4e4090e8479c5715e6ac, State: Initialized, Role: FOLLOWER
I20260924 05:04:45.104470 11084 consensus_queue.cc:260] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.104553 11086 consensus_queue.cc:260] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.104527 11085 raft_consensus.cc:348] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.104660 11085 raft_consensus.cc:374] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:04:45.104692 11085 raft_consensus.cc:729] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9474d5b9847f4a47908265e5e392efcb, State: Initialized, Role: FOLLOWER
I20260924 05:04:45.104748 11084 ts_tablet_manager.cc:1428] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:45.104773 11070 heartbeater.cc:503] Master 127.0.98.62:40091 was elected leader, sending a full tablet report...
I20260924 05:04:45.104823 11086 ts_tablet_manager.cc:1428] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:45.104791 11085 consensus_queue.cc:260] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.104944 10936 heartbeater.cc:503] Master 127.0.98.62:40091 was elected leader, sending a full tablet report...
I20260924 05:04:45.105023 11085 ts_tablet_manager.cc:1428] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:04:45.105161 10802 heartbeater.cc:503] Master 127.0.98.62:40091 was elected leader, sending a full tablet report...
I20260924 05:04:45.177356 11090 raft_consensus.cc:482] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:04:45.177464 11090 raft_consensus.cc:504] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.177805 11090 leader_election.cc:288] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e9f31a4b6e4e4090e8479c5715e6ac (127.0.98.2:45319), 9474d5b9847f4a47908265e5e392efcb (127.0.98.1:46291)
W20260924 05:04:45.178398 10937 tablet.cc:2369] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:04:45.181470 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.178110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34135 (local address 127.0.98.1:46291)
0924 05:04:45.178157 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.178159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.178228 (+ 69us) server_negotiation.cc:407] Connection header received
0924 05:04:45.178272 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.178273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.178283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.178336 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:45.178442 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.179162 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.180565 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.181147 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.181214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.181223 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.181361 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.181404 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:04:45.181497 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.177936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.98.3:48965)
0924 05:04:45.177983 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.177995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:45.178033 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:45.178186 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:45.178187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:45.178189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:45.178256 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.178257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.179375 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.179375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.180495 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.180497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.181215 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.181216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.181303 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.181305 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:04:45.181369 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:04:45.181398 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2}
I20260924 05:04:45.181504 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.178028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46291 (local address 127.0.98.3:34135)
0924 05:04:45.178166 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.178175 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:04:45.178225 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:04:45.178343 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:04:45.178344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:04:45.178345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:04:45.178391 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.178392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.179258 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.179259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.180495 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:04:45.180497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.181215 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.181216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:04:45.181300 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.181304 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:04:45.181355 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:04:45.181392 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260924 05:04:45.181545 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.177995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48965 (local address 127.0.98.2:45319)
0924 05:04:45.178035 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.178037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.178078 (+ 41us) server_negotiation.cc:407] Connection header received
0924 05:04:45.178106 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.178106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.178117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.178161 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:04:45.178651 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.179318 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.180565 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.181147 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.181214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.181223 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.181429 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.181492 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:04:45.181874 10756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5815041674a4e028f2b6d91f72f404e" candidate_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9474d5b9847f4a47908265e5e392efcb" is_pre_election: true
I20260924 05:04:45.181875 10890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5815041674a4e028f2b6d91f72f404e" candidate_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" is_pre_election: true
I20260924 05:04:45.182018 10756 raft_consensus.cc:2454] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d13424f23af4bb5ac18faa43e4c5e46 in term 0.
I20260924 05:04:45.182018 10890 raft_consensus.cc:2454] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d13424f23af4bb5ac18faa43e4c5e46 in term 0.
I20260924 05:04:45.182226 10958 leader_election.cc:302] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d13424f23af4bb5ac18faa43e4c5e46, 97e9f31a4b6e4e4090e8479c5715e6ac; no voters:
I20260924 05:04:45.182374 11090 raft_consensus.cc:2781] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:04:45.182431 11090 raft_consensus.cc:482] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:04:45.182459 11090 raft_consensus.cc:3037] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:45.183100 11090 raft_consensus.cc:504] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.183189 11090 leader_election.cc:288] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [CANDIDATE]: Term 1 election: Requested vote from peers 97e9f31a4b6e4e4090e8479c5715e6ac (127.0.98.2:45319), 9474d5b9847f4a47908265e5e392efcb (127.0.98.1:46291)
I20260924 05:04:45.183332 10890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5815041674a4e028f2b6d91f72f404e" candidate_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac"
I20260924 05:04:45.183331 10756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5815041674a4e028f2b6d91f72f404e" candidate_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9474d5b9847f4a47908265e5e392efcb"
I20260924 05:04:45.183395 10756 raft_consensus.cc:3037] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:45.183396 10890 raft_consensus.cc:3037] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:04:45.184017 10890 raft_consensus.cc:2454] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d13424f23af4bb5ac18faa43e4c5e46 in term 1.
I20260924 05:04:45.184088 10756 raft_consensus.cc:2454] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d13424f23af4bb5ac18faa43e4c5e46 in term 1.
I20260924 05:04:45.184185 10958 leader_election.cc:302] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d13424f23af4bb5ac18faa43e4c5e46, 97e9f31a4b6e4e4090e8479c5715e6ac; no voters:
I20260924 05:04:45.184258 11090 raft_consensus.cc:2781] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:04:45.184388 11090 raft_consensus.cc:686] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 1 LEADER]: Becoming Leader. State: Replica: 1d13424f23af4bb5ac18faa43e4c5e46, State: Running, Role: LEADER
I20260924 05:04:45.184468 11090 consensus_queue.cc:237] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } }
I20260924 05:04:45.185163 10611 catalog_manager.cc:5495] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d13424f23af4bb5ac18faa43e4c5e46 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d13424f23af4bb5ac18faa43e4c5e46" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 } health_report { overall_health: UNKNOWN } } }
I20260924 05:04:45.189540 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.187419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.0.98.3:41027)
0924 05:04:45.187468 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:04:45.187470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:04:45.187476 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:04:45.187502 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:04:45.187503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:04:45.187517 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:04:45.187572 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:04:45.187696 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.188135 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.188850 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:04:45.189049 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:04:45.189098 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:04:45.189241 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:04:45.189337 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:04:45.189403 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:04:45.189441 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 05:04:45.189460 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:04:45.190987 10890 raft_consensus.cc:1261] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 1 FOLLOWER]: Refusing update from remote peer 1d13424f23af4bb5ac18faa43e4c5e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:45.191001 10756 raft_consensus.cc:1261] T c5815041674a4e028f2b6d91f72f404e P 9474d5b9847f4a47908265e5e392efcb [term 1 FOLLOWER]: Refusing update from remote peer 1d13424f23af4bb5ac18faa43e4c5e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:04:45.191291 11094 consensus_queue.cc:1035] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e9f31a4b6e4e4090e8479c5715e6ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:45.191416 11090 consensus_queue.cc:1035] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9474d5b9847f4a47908265e5e392efcb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:04:45.193933 11098 mvcc.cc:205] Tried to move back new op lower bound from 7332766864140210176 to 7332766864116187136. Current Snapshot: MvccSnapshot[applied={T|T < 7332766864140210176 or (T in {7332766864140210176})}]
I20260924 05:04:45.194062 11097 mvcc.cc:205] Tried to move back new op lower bound from 7332766864140210176 to 7332766864116187136. Current Snapshot: MvccSnapshot[applied={T|T < 7332766864140210176 or (T in {7332766864140210176})}]
I20260924 05:04:45.194235 11099 mvcc.cc:205] Tried to move back new op lower bound from 7332766864140210176 to 7332766864116187136. Current Snapshot: MvccSnapshot[applied={T|T < 7332766864140210176}]
I20260924 05:04:45.195382 10610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:45.195473 10610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:04:45.196086 10610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21fe408b756d428892d22f50c2cd2656: Sending DeleteTablet for 3 replicas of tablet c5815041674a4e028f2b6d91f72f404e
I20260924 05:04:45.196399 10870 tablet_service.cc:1514] Processing DeleteTablet for tablet c5815041674a4e028f2b6d91f72f404e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59634
I20260924 05:04:45.196445 11004 tablet_service.cc:1514] Processing DeleteTablet for tablet c5815041674a4e028f2b6d91f72f404e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50144
I20260924 05:04:45.196578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10673
I20260924 05:04:45.196610 10736 tablet_service.cc:1514] Processing DeleteTablet for tablet c5815041674a4e028f2b6d91f72f404e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:04:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59606
I20260924 05:04:45.197118 11107 tablet_replica.cc:330] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: stopping tablet replica
I20260924 05:04:45.197189 11107 raft_consensus.cc:2229] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:04:45.197245 11107 raft_consensus.cc:2258] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:45.197350 11106 tablet_replica.cc:330] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: stopping tablet replica
I20260924 05:04:45.197424 11106 raft_consensus.cc:2229] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:04:45.197502 11106 raft_consensus.cc:2258] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:04:45.197667 11107 ts_tablet_manager.cc:1905] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:45.197893 11106 ts_tablet_manager.cc:1905] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:04:45.199195 11107 ts_tablet_manager.cc:1918] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:45.199213 11106 ts_tablet_manager.cc:1918] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:04:45.199252 11107 log.cc:1196] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/c5815041674a4e028f2b6d91f72f404e
I20260924 05:04:45.199257 11106 log.cc:1196] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/c5815041674a4e028f2b6d91f72f404e
I20260924 05:04:45.199498 11107 ts_tablet_manager.cc:1939] T c5815041674a4e028f2b6d91f72f404e P 1d13424f23af4bb5ac18faa43e4c5e46: Deleting consensus metadata
I20260924 05:04:45.199498 11106 ts_tablet_manager.cc:1939] T c5815041674a4e028f2b6d91f72f404e P 97e9f31a4b6e4e4090e8479c5715e6ac: Deleting consensus metadata
I20260924 05:04:45.199828 10596 catalog_manager.cc:4841] TS 97e9f31a4b6e4e4090e8479c5715e6ac (127.0.98.2:45319): tablet c5815041674a4e028f2b6d91f72f404e (table test-table [id=a8563ff4d4d04dfc9759e46ab0f0b9f6]) successfully deleted
I20260924 05:04:45.199908 10596 catalog_manager.cc:4841] TS 1d13424f23af4bb5ac18faa43e4c5e46 (127.0.98.3:41027): tablet c5815041674a4e028f2b6d91f72f404e (table test-table [id=a8563ff4d4d04dfc9759e46ab0f0b9f6]) successfully deleted
W20260924 05:04:45.203742 10596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46291 (error 108)
I20260924 05:04:45.203961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10807
I20260924 05:04:45.204139 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:04:45.204019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46291 (local address 127.0.0.1:59608)
0924 05:04:45.204082 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 05:04:45.204112 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 05:04:45.204474 10596 catalog_manager.cc:4584] TS 9474d5b9847f4a47908265e5e392efcb (127.0.98.1:46291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5815041674a4e028f2b6d91f72f404e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46291: connect: Connection refused (error 111)
I20260924 05:04:45.208225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10941
I20260924 05:04:45.211822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 10579
2026-09-24T05:04:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1752 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:04:45 dist-test-slave-m01t krb5kdc[11111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:04:45 dist-test-slave-m01t krb5kdc[11111](info): set up 2 sockets
Sep 24 05:04:45 dist-test-slave-m01t krb5kdc[11111](info): commencing operation
krb5kdc: starting...
W20260924 05:04:46.150171 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:04:46 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:04:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 05:04:46.289562 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyORENn/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/postgres -l logfile start
2026-09-24 05:04:48.410 UTC [11141] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 05:04:48.410 UTC [11141] LOG: listening on IPv4 address "127.0.98.20", port 58217
2026-09-24 05:04:48.413 UTC [11141] LOG: listening on Unix socket "/tmp/.s.PGSQL.58217"
2026-09-24 05:04:48.431 UTC [11144] LOG: database system was shut down at 2026-09-24 05:04:47 UTC
2026-09-24 05:04:48.434 UTC [11141] LOG: database system is ready to accept connections
I20260924 05:04:49.302047 392 mini_postgres.cc:96] Postgres bound to 58217
2026-09-24 05:04:49.307 UTC [11152] FATAL: database "slave" does not exist
127.0.98.20:58217 - accepting connections
I20260924 05:04:49.308051 392 mini_ranger.cc:161] Starting Ranger...
I20260924 05:04:49.321007 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 05:04:49.679103 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 05:04:49.679171 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-admin
2026-09-24 05:04:49,941 [I] DB FLAVOR :POSTGRES
2026-09-24 05:04:49,942 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:04:49,942 [I] Checking connection..
2026-09-24 05:04:49,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 05:04:50,184 [I] Checking connection passed.
2026-09-24 05:04:50,184 [I] --------- Verifying version history table ---------
2026-09-24 05:04:50,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:04:50,409 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:04:50,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:04:50,636 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:04:50,636 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 05:04:50,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 05:04:50.860 UTC [11216] WARNING: there is no transaction in progress
2026-09-24 05:04:50,865 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 05:04:50,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:04:51,111 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 05:04:51,111 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:04:51,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 05:04:51,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 05:04:51,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m01t.c.gcp-upstream.internal','N') ;"
2026-09-24 05:04:51,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:04:52,051 [I] Table x_portal_user does not exist in database ranger
2026-09-24 05:04:52,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:04:52,299 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 05:04:52,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:04:52,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:04:52,782 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 05:04:52,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 05:04:53.621 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.636 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.650 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.663 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.675 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.693 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.745 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.750 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.762 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.770 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.778 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.791 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.799 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.804 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.810 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:53.815 UTC [11324] WARNING: there is no transaction in progress
2026-09-24 05:04:54,329 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 05:04:54,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 05:04:54,563 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 05:04:54,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:04:54,801 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 05:04:54,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:04:55,049 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 05:04:55,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:04:55,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:04:55,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m01t.c.gcp-upstream.internal';"
2026-09-24 05:04:55,726 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 05:04:55,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 05:04:55,950 [I] DB_PATCHES have already been applied
I20260924 05:04:55.955415 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyORENn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 05:04:55.955470 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 05:04:55.958091 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:33173
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:04:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 05:04:56.949295 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.641s user 0.001s sys 0.006s
Sep 24, 2026 5:05:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:05:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 5:05:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:05:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:05:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 05:05:18.131294 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 05:05:18 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226318, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 05:05:18.201944 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 05:05:18.210310 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 05:05:18.920660 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 05:05:18.920733 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms
I20260924 05:05:18.920745 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58217
2026-09-24 05:05:19,081 [I] DB FLAVOR :POSTGRES
2026-09-24 05:05:19,082 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:05:19,082 [I] Checking connection
2026-09-24 05:05:19,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 05:05:19,331 [I] connection success
2026-09-24 05:05:19,331 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 05:05:19,331 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 05:05:19,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 05:05:19,556 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 05:05:19,556 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:05:19,556 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 05:05:19,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 05:05:19,813 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 05:05:20.267655 392 mini_ranger_kms.cc:310] Created kms service
I20260924 05:05:20.444234 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 05:05:20.484928 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 05:05:20 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226318, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:05:20.837752 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 05:05:20.837863 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 05:05:20.837883 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 05:05:20.841295 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11504
I20260924 05:05:20.841391 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37903
Sep 24, 2026 5:05:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:05:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:05:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:05:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:05:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 05:05:22.849310 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.647s user 0.008s sys 0.136s
I20260924 05:05:22.849424 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 05:05:22.849502 392 mini_ranger_kms.cc:401] 127.0.98.20:37903/kms/v1/keys
Sep 24 05:05:23 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226323, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 05:05:23 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226323, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:05:23 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226323, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:05:26 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:05:26.208609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45309
--ranger_config_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37903/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:05:26.288198 11555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:05:26.288409 11555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:05:26.288431 11555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:05:26.289584 11555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:05:26.289628 11555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:05:26.289640 11555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:05:26.289651 11555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:05:26.289661 11555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:05:26.291656 11555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45309
--ranger_config_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:05:26.291885 11555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:05:26.292109 11555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:05:26.296842 11560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:05:26.296952 11555 server_base.cc:1034] running on GCE node
W20260924 05:05:26.296850 11561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:26.296837 11563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:05:26.297336 11555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:05:26.297638 11555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:05:26.298779 11555 hybrid_clock.cc:648] HybridClock initialized: now 1790226326298763 us; error 32 us; skew 500 ppm
Sep 24 05:05:26 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:05:26.301383 11555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:05:26.301967 11555 webserver.cc:460] Webserver started at http://127.0.98.62:35847/ using document root <none> and password file <none>
I20260924 05:05:26.302232 11555 fs_manager.cc:359] Metadata directory not provided
I20260924 05:05:26.302306 11555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:05:26.302466 11555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:05:26 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:05:26.354591 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "07154bfad645486d87a83696b767ebc7"
format_stamp: "Formatted at 2026-09-24 05:05:26 on dist-test-slave-m01t"
server_key: "bee143415d7e90bc467d0e392cd103d2"
server_key_iv: "6ecf3f79141bb165732c8d277dc8c5e7"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:26.355051 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "07154bfad645486d87a83696b767ebc7"
format_stamp: "Formatted at 2026-09-24 05:05:26 on dist-test-slave-m01t"
server_key: "bee143415d7e90bc467d0e392cd103d2"
server_key_iv: "6ecf3f79141bb165732c8d277dc8c5e7"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:26.356752 11555 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 05:05:26.372164 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:26.372393 11555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:05:26.372480 11555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "07154bfad645486d87a83696b767ebc7"
format_stamp: "Formatted at 2026-09-24 05:05:26 on dist-test-slave-m01t"
server_key: "bee143415d7e90bc467d0e392cd103d2"
server_key_iv: "6ecf3f79141bb165732c8d277dc8c5e7"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:26.372586 11555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:05:26.390230 11555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:05:26.391933 11555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:05:26.392072 11555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:05:26.396725 11555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45309
I20260924 05:05:26.396765 11622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45309 every 8 connection(s)
I20260924 05:05:26.397095 11555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:05:26.397842 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:05:26.400846 11623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7: Bootstrap starting.
I20260924 05:05:26.402079 11623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7: Neither blocks nor log segments found. Creating new log.
I20260924 05:05:26.402370 11623 log.cc:824] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:05:26.403110 11623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7: No bootstrap required, opened a new log
I20260924 05:05:26.404162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 11555
I20260924 05:05:26.404250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:05:26.404412 392 external_mini_cluster.cc:1428] Setting key 94cb696b7754ba966c57241306fb29f8
I20260924 05:05:26.405870 11623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07154bfad645486d87a83696b767ebc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 05:05:26.406127 11623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:05:26.406183 11623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07154bfad645486d87a83696b767ebc7, State: Initialized, Role: FOLLOWER
I20260924 05:05:26.406328 11623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07154bfad645486d87a83696b767ebc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 05:05:26.406426 11623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:05:26.406486 11623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:05:26.406555 11623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:05:26.407256 11623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07154bfad645486d87a83696b767ebc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 05:05:26.407370 11623 leader_election.cc:302] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07154bfad645486d87a83696b767ebc7; no voters:
I20260924 05:05:26.407533 11623 leader_election.cc:288] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:05:26.407610 11628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:05:26.407848 11623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:05:26.408599 11628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [term 1 LEADER]: Becoming Leader. State: Replica: 07154bfad645486d87a83696b767ebc7, State: Running, Role: LEADER
I20260924 05:05:26.408753 11628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07154bfad645486d87a83696b767ebc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 05:05:26.409106 11628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07154bfad645486d87a83696b767ebc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07154bfad645486d87a83696b767ebc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260924 05:05:26.409235 11628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:05:26.409332 11623 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:05:26.409392 11628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07154bfad645486d87a83696b767ebc7. Latest consensus state: current_term: 1 leader_uuid: "07154bfad645486d87a83696b767ebc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07154bfad645486d87a83696b767ebc7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260924 05:05:26.409439 11628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:05:26.409695 11642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:05:26.410197 11642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:05:26 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:05:26.417423 11642 catalog_manager.cc:1347] Generated new cluster ID: 09b5df099d5146d9b316266092bb6672
I20260924 05:05:26.417552 11642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:05:26.422602 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:26.404867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.0.98.62:45309)
0924 05:05:26.405034 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:26.405043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:26.405073 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:05:26.405814 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:26.405821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:26.405871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:26.405994 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:05:26.406674 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:26.407694 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:26.408527 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:26.410245 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:26.410370 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:26.412940 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:05:26.412988 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:05:26.412988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:05:26.413084 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:05:26.416228 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:26.416631 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:26.416634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:26.416635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:26.416762 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:26.416962 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:26.416963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:26.416963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:26.417394 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:05:26.417522 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:26.422154 (+ 4632us) server_negotiation.cc:299] Negotiation successful
0924 05:05:26.422332 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260924 05:05:26.438019 11642 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:05:26.438869 11642 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:05:26.442695 11642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7: Generated new TSK 0
I20260924 05:05:26.442948 11642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:05:27.178140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:05:27.284713 11668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:05:27.284935 11668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:05:27.284967 11668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:05:27.286132 11668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:05:27.286173 11668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:05:27.286208 11668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:05:27.287640 11668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:05:27.287870 11668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:05:27.288082 11668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:05:27.293984 11668 server_base.cc:1034] running on GCE node
W20260924 05:05:27.293987 11673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:27.294230 11674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:27.294287 11676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:05:27.294539 11668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:05:27.294831 11668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:05:27.295976 11668 hybrid_clock.cc:648] HybridClock initialized: now 1790226327295958 us; error 33 us; skew 500 ppm
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:05:27.299168 11668 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:05:27.299685 11668 webserver.cc:460] Webserver started at http://127.0.98.1:39181/ using document root <none> and password file <none>
I20260924 05:05:27.299927 11668 fs_manager.cc:359] Metadata directory not provided
I20260924 05:05:27.299986 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:05:27.300158 11668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:05:27.316221 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "fcaed45df3cf42acb8df736c61ff8f71"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "5bd8e1f2c1e4abf91ff5c01951627a01"
server_key_iv: "9bc8f6df195775c1e5bb48ec696b7949"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.316677 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fcaed45df3cf42acb8df736c61ff8f71"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "5bd8e1f2c1e4abf91ff5c01951627a01"
server_key_iv: "9bc8f6df195775c1e5bb48ec696b7949"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.322883 11668 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260924 05:05:27.336992 11683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.337251 11668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:05:27.337365 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "fcaed45df3cf42acb8df736c61ff8f71"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "5bd8e1f2c1e4abf91ff5c01951627a01"
server_key_iv: "9bc8f6df195775c1e5bb48ec696b7949"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.337467 11668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:05:27.355937 11668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:05:27.358458 11668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:05:27.358677 11668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:05:27.358978 11668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:05:27.359395 11668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:05:27.359477 11668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.359570 11668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:05:27.359642 11668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.371265 11668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41835
I20260924 05:05:27.371833 11668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:05:27.377079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 11668
I20260924 05:05:27.377194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:05:27.377377 392 external_mini_cluster.cc:1428] Setting key 71f2cbd8ebce81d335dfea337b48502b
I20260924 05:05:27.382041 11798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41835 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:05:27.408384 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.382820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:59177)
0924 05:05:27.393849 (+ 11029us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:27.393928 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:27.394623 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:27.395023 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:27.395028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:27.395294 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:05:27.395594 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.395600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.397844 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.397845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.399064 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.399067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.401960 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.401961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.402122 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.402878 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:05:27.402890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:05:27.404553 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:05:27.406627 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:05:27.406635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:05:27.406637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:05:27.406941 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:05:27.407161 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:05:27.407162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:05:27.407162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:05:27.407302 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:05:27.407646 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:05:27.407652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:05:27.407834 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 05:05:27.407949 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 05:05:27.408122 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10959,"thread_start_us":94,"threads_started":1}
I20260924 05:05:27.409005 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.383468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59177 (local address 127.0.98.62:45309)
0924 05:05:27.394800 (+ 11332us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.394802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.394809 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:05:27.394829 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.394830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.394840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.394895 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:05:27.395709 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.396501 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.401450 (+ 4949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.401802 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.401882 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.404687 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:05:27.404689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:05:27.404689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:05:27.404697 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:05:27.406530 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.407029 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.407030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.407031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.407094 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.407413 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.407414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.407414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.407559 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:05:27.407602 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.408814 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.408902 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11302,"thread_start_us":191,"threads_started":1}
I20260924 05:05:27.409786 11799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260924 05:05:27.409886 11799 heartbeater.cc:464] Registering TS with master...
I20260924 05:05:27.410074 11799 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260924 05:05:27.410715 11587 ts_manager.cc:194] Registered new tserver with Master: fcaed45df3cf42acb8df736c61ff8f71 (127.0.98.1:41835)
I20260924 05:05:27.411618 11587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:05:27.414652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:05:27.533491 11805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:05:27.533890 11805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:05:27.533987 11805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:05:27.535851 11805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:05:27.535962 11805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:05:27.536046 11805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:05:27.538358 11805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:05:27.538846 11805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:05:27.539183 11805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:05:27.545854 11815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:27.545861 11813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:27.546109 11812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:05:27.546857 11805 server_base.cc:1034] running on GCE node
I20260924 05:05:27.547075 11805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:05:27.547353 11805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:05:27.548528 11805 hybrid_clock.cc:648] HybridClock initialized: now 1790226327548511 us; error 29 us; skew 500 ppm
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:05:27.551903 11805 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:05:27.552459 11805 webserver.cc:460] Webserver started at http://127.0.98.2:46191/ using document root <none> and password file <none>
I20260924 05:05:27.552721 11805 fs_manager.cc:359] Metadata directory not provided
I20260924 05:05:27.552779 11805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:05:27.552959 11805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:05:27.570576 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "89beeb0764d1427b86a73b64328ac7bb"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "d22b2db1524909e6892240be92163163"
server_key_iv: "4d47a331f0b28b060b3dd51f4dc01c53"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.582006 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "89beeb0764d1427b86a73b64328ac7bb"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "d22b2db1524909e6892240be92163163"
server_key_iv: "4d47a331f0b28b060b3dd51f4dc01c53"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.584059 11805 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:05:27.599026 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.599435 11805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:05:27.599542 11805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "89beeb0764d1427b86a73b64328ac7bb"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "d22b2db1524909e6892240be92163163"
server_key_iv: "4d47a331f0b28b060b3dd51f4dc01c53"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.599637 11805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:05:27.627931 11805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:05:27.630560 11805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:05:27.630774 11805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:05:27.631090 11805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:05:27.631492 11805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:05:27.631585 11805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.631656 11805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:05:27.631711 11805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.638785 11805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42207
I20260924 05:05:27.639262 11805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:05:27.640317 11936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42207 every 8 connection(s)
I20260924 05:05:27.643430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 11805
I20260924 05:05:27.643536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:05:27.643728 392 external_mini_cluster.cc:1428] Setting key f801079b786323cca3086a94b83c1b49
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:05:27.652204 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.641335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53141 (local address 127.0.98.62:45309)
0924 05:05:27.641384 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.641386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.641704 (+ 318us) server_negotiation.cc:407] Connection header received
0924 05:05:27.642439 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.642441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.642453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.642485 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:05:27.643343 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.644102 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.645707 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.646058 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.646109 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.648709 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:05:27.648711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:05:27.648711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:05:27.648720 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:05:27.650636 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.651121 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.651122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.651123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.651178 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.651441 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.651442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.651442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.651577 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:05:27.651617 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.652019 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.652107 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:05:27.653107 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.641043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:53141)
0924 05:05:27.641567 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:27.641614 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:27.642307 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:27.642674 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:27.642679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:27.642945 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:05:27.643230 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.643237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.644301 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.644302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.645601 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.645604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.646200 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.646200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.646347 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.646951 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:05:27.646961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:05:27.648596 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:05:27.650730 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:05:27.650736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:05:27.650739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:05:27.651033 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:05:27.651243 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:05:27.651244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:05:27.651244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:05:27.651370 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:05:27.651661 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:05:27.651667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:05:27.651858 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 05:05:27.652707 (+ 849us) client_negotiation.cc:240] Negotiation successful
0924 05:05:27.652888 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":89,"threads_started":1}
I20260924 05:05:27.653858 11937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260924 05:05:27.653954 11937 heartbeater.cc:464] Registering TS with master...
I20260924 05:05:27.654137 11937 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260924 05:05:27.654547 11587 ts_manager.cc:194] Registered new tserver with Master: 89beeb0764d1427b86a73b64328ac7bb (127.0.98.2:42207)
I20260924 05:05:27.654910 11587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53141
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:05:27.685555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:05:27.795827 11942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:05:27.796131 11942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:05:27.796168 11942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:05:27.797998 11942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:05:27.798105 11942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:05:27.798162 11942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:05:27.800336 11942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:05:27.800582 11942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:05:27.800894 11942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:05:27.807179 11947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:27.807551 11948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:05:27.807183 11950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:05:27.807858 11942 server_base.cc:1034] running on GCE node
I20260924 05:05:27.808089 11942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:05:27.808370 11942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:05:27.817824 11942 hybrid_clock.cc:648] HybridClock initialized: now 1790226327817783 us; error 39 us; skew 500 ppm
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:05:27.821115 11942 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:05:27.821625 11942 webserver.cc:460] Webserver started at http://127.0.98.3:43601/ using document root <none> and password file <none>
I20260924 05:05:27.821871 11942 fs_manager.cc:359] Metadata directory not provided
I20260924 05:05:27.821939 11942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:05:27.822067 11942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:05:27.835150 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "88e236649264453391fe9d368e877e3b"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "98732130ed0e2e6f3e74e23037a7e3dc"
server_key_iv: "a9ee85e5c6a285d0d870a77a1424c6b0"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.835646 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88e236649264453391fe9d368e877e3b"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "98732130ed0e2e6f3e74e23037a7e3dc"
server_key_iv: "a9ee85e5c6a285d0d870a77a1424c6b0"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.837560 11942 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 05:05:27.865881 11957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.866143 11942 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260924 05:05:27.866246 11942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "88e236649264453391fe9d368e877e3b"
format_stamp: "Formatted at 2026-09-24 05:05:27 on dist-test-slave-m01t"
server_key: "98732130ed0e2e6f3e74e23037a7e3dc"
server_key_iv: "a9ee85e5c6a285d0d870a77a1424c6b0"
server_key_version: "kuduclusterkey@0"
I20260924 05:05:27.866345 11942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:05:27.887704 11942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:05:27.890604 11942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:05:27.890825 11942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:05:27.891096 11942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:05:27.891508 11942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:05:27.891587 11942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.891649 11942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:05:27.891692 11942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:05:27.897794 11942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44325
I20260924 05:05:27.898258 11942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:05:27.901789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 11942
I20260924 05:05:27.901917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:05:27.902103 392 external_mini_cluster.cc:1428] Setting key b2590b1ac7240445145ec81a1d8dc9f6
I20260924 05:05:27.909001 12070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44325 every 8 connection(s)
Sep 24 05:05:27 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:05:27.932976 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.909845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48737 (local address 127.0.98.62:45309)
0924 05:05:27.909888 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.909891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.922019 (+ 12128us) server_negotiation.cc:407] Connection header received
0924 05:05:27.922716 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.922718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.922731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.922781 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:05:27.923638 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.924431 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.926552 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.926850 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.926896 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.929706 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:05:27.929738 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:05:27.929738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:05:27.929746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:05:27.931405 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.931895 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.931896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.931897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.931953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.932203 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.932203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.932204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.932333 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:05:27.932377 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.932781 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.932880 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:05:27.933836 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.909904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:48737)
0924 05:05:27.921845 (+ 11941us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:27.921906 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:27.922603 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:27.922937 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:27.922943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:27.923222 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:05:27.923520 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.923527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.924824 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.924825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.926450 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.926453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.926989 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.926990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.927138 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.927843 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:05:27.927854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:05:27.929591 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:05:27.931498 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:05:27.931505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:05:27.931508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:05:27.931804 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:05:27.932013 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:05:27.932014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:05:27.932014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:05:27.932140 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:05:27.932421 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:05:27.932428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:05:27.932620 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 05:05:27.933354 (+ 734us) client_negotiation.cc:240] Negotiation successful
0924 05:05:27.933537 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11877,"thread_start_us":110,"threads_started":1}
I20260924 05:05:27.934527 12071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260924 05:05:27.934650 12071 heartbeater.cc:464] Registering TS with master...
I20260924 05:05:27.934864 12071 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260924 05:05:27.935389 11587 ts_manager.cc:194] Registered new tserver with Master: 88e236649264453391fe9d368e877e3b (127.0.98.3:44325)
I20260924 05:05:27.935865 11587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48737
I20260924 05:05:27.937173 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:05:27.957934 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.946088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.0.98.62:45309)
0924 05:05:27.946137 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.946139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.949956 (+ 3817us) server_negotiation.cc:407] Connection header received
0924 05:05:27.950077 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.950079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.950092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.950126 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:05:27.950977 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.951778 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.952892 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.953211 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.953252 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.954966 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:05:27.954968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:05:27.954968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:05:27.954976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:05:27.956611 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.957022 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.957023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.957024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.957086 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:05:27.957281 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:05:27.957282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:05:27.957282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:05:27.957411 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:05:27.957487 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.957696 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.957833 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:05:27.960139 11587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:05:27.962464 11587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:05:27.975589 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.965835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41835 (local address 127.0.0.1:58500)
0924 05:05:27.968436 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:27.968444 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:27.968483 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:27.969280 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:27.969284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:27.969287 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:05:27.969368 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.969370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.971137 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.971138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.972872 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.972876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.973979 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.973980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.975352 (+ 1372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.975378 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 05:05:27.975443 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 05:05:27.975500 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"thread_start_us":68,"threads_started":1}
I20260924 05:05:27.976069 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.966225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.0.98.1:41835)
0924 05:05:27.968152 (+ 1927us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.968158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.968534 (+ 376us) server_negotiation.cc:407] Connection header received
0924 05:05:27.968573 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.968576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.968594 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.969183 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:05:27.969450 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.971048 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.972969 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.973808 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.973886 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.973947 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.975947 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.975997 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":55,"threads_started":1}
I20260924 05:05:27.976955 11733 tablet_service.cc:1467] Processing CreateTablet for tablet 081a379a3e474842b5941e48783f3e60 (DEFAULT_TABLE table=test-table [id=47fdd39784aa46e8a90a027cf498ad6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:05:27.977278 11733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081a379a3e474842b5941e48783f3e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:05:27.981684 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.965992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42207 (local address 127.0.0.1:60442)
0924 05:05:27.970552 (+ 4560us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:27.970563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:27.970613 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:27.971316 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:27.971317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:27.971319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:05:27.977820 (+ 6501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.977825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.979167 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.979169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.980264 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.980267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.981416 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.981418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.981487 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.981492 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:05:27.981550 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:05:27.981594 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4541,"mutex_wait_us":6426,"thread_start_us":49,"threads_started":1}
I20260924 05:05:27.982277 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.966101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.0.98.2:42207)
0924 05:05:27.971060 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.971065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.971078 (+ 13us) server_negotiation.cc:407] Connection header received
0924 05:05:27.971117 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.971119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.971134 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.971229 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:05:27.977963 (+ 6734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.979067 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.980622 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.981323 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.981394 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.981466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.982163 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.982202 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4916,"thread_start_us":87,"threads_started":1}
I20260924 05:05:27.983040 11858 tablet_service.cc:1467] Processing CreateTablet for tablet 081a379a3e474842b5941e48783f3e60 (DEFAULT_TABLE table=test-table [id=47fdd39784aa46e8a90a027cf498ad6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:05:27.983384 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081a379a3e474842b5941e48783f3e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:05:27.984992 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.965573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44325 (local address 127.0.0.1:46344)
0924 05:05:27.968223 (+ 2650us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:27.968271 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:27.968337 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:27.981524 (+ 13187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:27.981526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:27.981529 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:05:27.981582 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.981584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.982864 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.982865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.983929 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:27.983932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.984786 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.984787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:27.984853 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.984858 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:05:27.984881 (+ 23us) client_negotiation.cc:240] Negotiation successful
0924 05:05:27.984926 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2619,"thread_start_us":73,"threads_started":1}
I20260924 05:05:27.985296 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:27.965687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.0.98.3:44325)
0924 05:05:27.980445 (+ 14758us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:27.980449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:27.980459 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:05:27.980496 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:27.980498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:27.980514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:27.981649 (+ 1135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:05:27.981737 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.982768 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.984038 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:27.984700 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:27.985088 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:27.985169 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:27.985192 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 05:05:27.985227 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14712,"thread_start_us":105,"threads_started":1}
I20260924 05:05:27.985874 12005 tablet_service.cc:1467] Processing CreateTablet for tablet 081a379a3e474842b5941e48783f3e60 (DEFAULT_TABLE table=test-table [id=47fdd39784aa46e8a90a027cf498ad6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:05:27.986181 12005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081a379a3e474842b5941e48783f3e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:05:27.993896 12091 tablet_bootstrap.cc:492] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Bootstrap starting.
I20260924 05:05:27.993927 12092 tablet_bootstrap.cc:492] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Bootstrap starting.
I20260924 05:05:27.995316 12091 tablet_bootstrap.cc:654] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Neither blocks nor log segments found. Creating new log.
I20260924 05:05:27.995594 12092 tablet_bootstrap.cc:654] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Neither blocks nor log segments found. Creating new log.
I20260924 05:05:27.995726 12091 log.cc:824] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:05:27.995954 12092 log.cc:824] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:05:27.996387 12090 tablet_bootstrap.cc:492] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: Bootstrap starting.
I20260924 05:05:27.996925 12090 tablet_bootstrap.cc:654] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: Neither blocks nor log segments found. Creating new log.
I20260924 05:05:27.997152 12090 log.cc:824] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: Log is configured to *not* fsync() on all Append() calls
I20260924 05:05:27.997898 12090 tablet_bootstrap.cc:492] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: No bootstrap required, opened a new log
I20260924 05:05:27.998016 12090 ts_tablet_manager.cc:1397] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:05:28.002184 12092 tablet_bootstrap.cc:492] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: No bootstrap required, opened a new log
I20260924 05:05:28.002830 12092 ts_tablet_manager.cc:1397] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 05:05:28.004927 12092 raft_consensus.cc:348] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.005227 12092 raft_consensus.cc:374] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:05:28.005306 12092 raft_consensus.cc:729] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e236649264453391fe9d368e877e3b, State: Initialized, Role: FOLLOWER
I20260924 05:05:28.005528 12092 consensus_queue.cc:260] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.005944 12092 ts_tablet_manager.cc:1428] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:05:28.006103 12071 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260924 05:05:28.001575 12090 raft_consensus.cc:348] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.007262 12090 raft_consensus.cc:374] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:05:28.007367 12090 raft_consensus.cc:729] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcaed45df3cf42acb8df736c61ff8f71, State: Initialized, Role: FOLLOWER
I20260924 05:05:28.007517 12091 tablet_bootstrap.cc:492] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: No bootstrap required, opened a new log
I20260924 05:05:28.007609 12091 ts_tablet_manager.cc:1397] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260924 05:05:28.007611 12090 consensus_queue.cc:260] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.007982 12090 ts_tablet_manager.cc:1428] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260924 05:05:28.008159 11799 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260924 05:05:28.009814 12091 raft_consensus.cc:348] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.010033 12091 raft_consensus.cc:374] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:05:28.010120 12091 raft_consensus.cc:729] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89beeb0764d1427b86a73b64328ac7bb, State: Initialized, Role: FOLLOWER
I20260924 05:05:28.010257 12091 consensus_queue.cc:260] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.010543 11937 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260924 05:05:28.010757 12091 ts_tablet_manager.cc:1428] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Sep 24 05:05:28 dist-test-slave-m01t krb5kdc[11111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:05:28.126940 12098 raft_consensus.cc:482] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:05:28.127096 12098 raft_consensus.cc:504] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.127514 12098 leader_election.cc:288] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcaed45df3cf42acb8df736c61ff8f71 (127.0.98.1:41835), 88e236649264453391fe9d368e877e3b (127.0.98.3:44325)
I20260924 05:05:28.131455 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:28.127709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41835 (local address 127.0.98.2:55179)
0924 05:05:28.127824 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:28.127837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:28.127884 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:28.128194 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:28.128196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:28.128198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:05:28.128263 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:28.128265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.129147 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.129148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:28.130391 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:28.130394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.131176 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.131177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:28.131251 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:28.131255 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:05:28.131296 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:05:28.131351 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":7}
I20260924 05:05:28.131579 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:28.127948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58395 (local address 127.0.98.3:44325)
0924 05:05:28.127987 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:28.127989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:28.128116 (+ 127us) server_negotiation.cc:407] Connection header received
0924 05:05:28.128210 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:28.128211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:28.128224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:28.128251 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:05:28.128493 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.129208 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.130637 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.131178 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.131228 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:28.131239 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:28.131460 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 05:05:28.131507 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:05:28.131791 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:28.127781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44325 (local address 127.0.98.2:58395)
0924 05:05:28.128056 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:28.128066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:05:28.128142 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:05:28.128312 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:05:28.128312 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:05:28.128328 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:05:28.128399 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:28.128401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.129333 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.129334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:28.130530 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:05:28.130533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.131299 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.131300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:05:28.131368 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:28.131372 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:05:28.131666 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 05:05:28.131708 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":71,"threads_started":1}
I20260924 05:05:28.132112 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:28.127985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55179 (local address 127.0.98.1:41835)
0924 05:05:28.128043 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:28.128045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:28.128053 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:05:28.128080 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:28.128081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:28.128094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:28.128146 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:05:28.128357 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.129055 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.130505 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.131076 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.131136 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:28.131148 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:28.132000 (+ 852us) server_negotiation.cc:299] Negotiation successful
0924 05:05:28.132045 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:05:28.132220 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a379a3e474842b5941e48783f3e60" candidate_uuid: "89beeb0764d1427b86a73b64328ac7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88e236649264453391fe9d368e877e3b" is_pre_election: true
I20260924 05:05:28.132387 12025 raft_consensus.cc:2454] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89beeb0764d1427b86a73b64328ac7bb in term 0.
I20260924 05:05:28.132369 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a379a3e474842b5941e48783f3e60" candidate_uuid: "89beeb0764d1427b86a73b64328ac7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcaed45df3cf42acb8df736c61ff8f71" is_pre_election: true
I20260924 05:05:28.132501 11739 raft_consensus.cc:2454] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89beeb0764d1427b86a73b64328ac7bb in term 0.
I20260924 05:05:28.132622 11824 leader_election.cc:302] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88e236649264453391fe9d368e877e3b, 89beeb0764d1427b86a73b64328ac7bb; no voters:
I20260924 05:05:28.132920 12098 raft_consensus.cc:2781] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:05:28.132999 12098 raft_consensus.cc:482] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:05:28.133024 12098 raft_consensus.cc:3037] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:05:28.133942 12098 raft_consensus.cc:504] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.134081 12098 leader_election.cc:288] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [CANDIDATE]: Term 1 election: Requested vote from peers fcaed45df3cf42acb8df736c61ff8f71 (127.0.98.1:41835), 88e236649264453391fe9d368e877e3b (127.0.98.3:44325)
W20260924 05:05:28.134084 11800 tablet.cc:2369] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:05:28.134344 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a379a3e474842b5941e48783f3e60" candidate_uuid: "89beeb0764d1427b86a73b64328ac7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcaed45df3cf42acb8df736c61ff8f71"
I20260924 05:05:28.134433 11739 raft_consensus.cc:3037] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:05:28.135257 11739 raft_consensus.cc:2454] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89beeb0764d1427b86a73b64328ac7bb in term 1.
I20260924 05:05:28.135548 11823 leader_election.cc:302] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89beeb0764d1427b86a73b64328ac7bb, fcaed45df3cf42acb8df736c61ff8f71; no voters:
I20260924 05:05:28.135823 12025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081a379a3e474842b5941e48783f3e60" candidate_uuid: "89beeb0764d1427b86a73b64328ac7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88e236649264453391fe9d368e877e3b"
I20260924 05:05:28.135906 12025 raft_consensus.cc:3037] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:05:28.136760 12025 raft_consensus.cc:2454] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89beeb0764d1427b86a73b64328ac7bb in term 1.
I20260924 05:05:28.137039 12098 raft_consensus.cc:2781] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:05:28.137197 12098 raft_consensus.cc:686] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 1 LEADER]: Becoming Leader. State: Replica: 89beeb0764d1427b86a73b64328ac7bb, State: Running, Role: LEADER
I20260924 05:05:28.137293 12098 consensus_queue.cc:237] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } }
I20260924 05:05:28.138299 11587 catalog_manager.cc:5495] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 89beeb0764d1427b86a73b64328ac7bb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "89beeb0764d1427b86a73b64328ac7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89beeb0764d1427b86a73b64328ac7bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 } health_report { overall_health: UNKNOWN } } }
W20260924 05:05:28.141817 11938 tablet.cc:2369] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:05:28.159834 12072 tablet.cc:2369] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:05:28.161983 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:28.158516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.98.2:42207)
0924 05:05:28.158574 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:05:28.158576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:05:28.158585 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:05:28.158625 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:05:28.158626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:05:28.158647 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:05:28.158716 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:05:28.158907 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.159654 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.160815 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:05:28.161099 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:05:28.161165 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:05:28.161365 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:05:28.161523 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:05:28.161604 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 05:05:28.161876 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 05:05:28.161898 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:05:28.164266 12025 raft_consensus.cc:1261] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 1 FOLLOWER]: Refusing update from remote peer 89beeb0764d1427b86a73b64328ac7bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:05:28.164706 12100 consensus_queue.cc:1035] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e236649264453391fe9d368e877e3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:05:28.166662 11739 raft_consensus.cc:1261] T 081a379a3e474842b5941e48783f3e60 P fcaed45df3cf42acb8df736c61ff8f71 [term 1 FOLLOWER]: Refusing update from remote peer 89beeb0764d1427b86a73b64328ac7bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:05:28.167042 12100 consensus_queue.cc:1035] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcaed45df3cf42acb8df736c61ff8f71" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:05:28.170951 12103 mvcc.cc:205] Tried to move back new op lower bound from 7332767040157806592 to 7332767040051101696. Current Snapshot: MvccSnapshot[applied={T|T < 7332767040157806592}]
I20260924 05:05:28.171633 12106 mvcc.cc:205] Tried to move back new op lower bound from 7332767040157806592 to 7332767040051101696. Current Snapshot: MvccSnapshot[applied={T|T < 7332767040157806592}]
I20260924 05:05:28.174357 11587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:05:28.174463 11587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:05:28.175282 11587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07154bfad645486d87a83696b767ebc7: Sending DeleteTablet for 3 replicas of tablet 081a379a3e474842b5941e48783f3e60
I20260924 05:05:28.175649 12005 tablet_service.cc:1514] Processing DeleteTablet for tablet 081a379a3e474842b5941e48783f3e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:05:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46344
I20260924 05:05:28.176085 11733 tablet_service.cc:1514] Processing DeleteTablet for tablet 081a379a3e474842b5941e48783f3e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:05:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58500
I20260924 05:05:28.176290 11858 tablet_service.cc:1514] Processing DeleteTablet for tablet 081a379a3e474842b5941e48783f3e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:05:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60442
I20260924 05:05:28.176805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 11668
I20260924 05:05:28.181754 12104 mvcc.cc:205] Tried to move back new op lower bound from 7332767040157806592 to 7332767040051101696. Current Snapshot: MvccSnapshot[applied={T|T < 7332767040157806592}]
I20260924 05:05:28.189200 12113 tablet_replica.cc:330] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: stopping tablet replica
I20260924 05:05:28.189473 12113 raft_consensus.cc:2229] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:05:28.189644 12113 raft_consensus.cc:2258] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:05:28.189813 12111 tablet_replica.cc:330] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: stopping tablet replica
I20260924 05:05:28.190063 12111 raft_consensus.cc:2229] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:05:28.190280 12111 raft_consensus.cc:2258] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 05:05:28.190650 11571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41835 (error 108)
I20260924 05:05:28.190879 12113 ts_tablet_manager.cc:1905] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:05:28.191464 12111 ts_tablet_manager.cc:1905] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:05:28.192567 12113 ts_tablet_manager.cc:1918] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:05:28.192648 12113 log.cc:1196] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/081a379a3e474842b5941e48783f3e60
I20260924 05:05:28.192899 12113 ts_tablet_manager.cc:1939] T 081a379a3e474842b5941e48783f3e60 P 89beeb0764d1427b86a73b64328ac7bb: Deleting consensus metadata
I20260924 05:05:28.193349 11571 catalog_manager.cc:4841] TS 89beeb0764d1427b86a73b64328ac7bb (127.0.98.2:42207): tablet 081a379a3e474842b5941e48783f3e60 (table test-table [id=47fdd39784aa46e8a90a027cf498ad6f]) successfully deleted
I20260924 05:05:28.193496 12111 ts_tablet_manager.cc:1918] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:05:28.193612 12111 log.cc:1196] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/081a379a3e474842b5941e48783f3e60
I20260924 05:05:28.194159 12111 ts_tablet_manager.cc:1939] T 081a379a3e474842b5941e48783f3e60 P 88e236649264453391fe9d368e877e3b: Deleting consensus metadata
I20260924 05:05:28.194598 11572 catalog_manager.cc:4841] TS 88e236649264453391fe9d368e877e3b (127.0.98.3:44325): tablet 081a379a3e474842b5941e48783f3e60 (table test-table [id=47fdd39784aa46e8a90a027cf498ad6f]) successfully deleted
I20260924 05:05:28.195058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 11805
I20260924 05:05:28.200609 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:05:28.197178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41835 (local address 127.0.0.1:58502)
0924 05:05:28.200513 (+ 3335us) negotiation.cc:107] Waiting for socket to connect
0924 05:05:28.200569 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3307}
W20260924 05:05:28.200762 11571 catalog_manager.cc:4584] TS fcaed45df3cf42acb8df736c61ff8f71 (127.0.98.1:41835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 081a379a3e474842b5941e48783f3e60: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41835: connect: Connection refused (error 111)
I20260924 05:05:28.201462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 11942
I20260924 05:05:28.206482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 11555
I20260924 05:05:28.211025 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 05:05:28.545188 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 05:05:28.545293 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 05:05:28.878931 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 05:05:28.879 UTC [11141] LOG: received smart shutdown request
2026-09-24 05:05:28.881 UTC [11141] LOG: background worker "logical replication launcher" (PID 11150) exited with exit code 1
2026-09-24 05:05:28.882 UTC [11145] LOG: shutting down
2026-09-24 05:05:29.014 UTC [11141] LOG: database system is shut down
2026-09-24T05:05:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43887 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:05:29 dist-test-slave-m01t krb5kdc[12124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:05:29 dist-test-slave-m01t krb5kdc[12124](info): set up 2 sockets
Sep 24 05:05:29 dist-test-slave-m01t krb5kdc[12124](info): commencing operation
krb5kdc: starting...
W20260924 05:05:30.041357 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:05:30 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:05:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 05:05:30.190274 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyORENn/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/postgres -l logfile start
2026-09-24 05:05:32.331 UTC [12154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 05:05:32.331 UTC [12154] LOG: listening on IPv4 address "127.0.98.20", port 56127
2026-09-24 05:05:32.334 UTC [12154] LOG: listening on Unix socket "/tmp/.s.PGSQL.56127"
2026-09-24 05:05:32.360 UTC [12157] LOG: database system was shut down at 2026-09-24 05:05:31 UTC
2026-09-24 05:05:32.363 UTC [12154] LOG: database system is ready to accept connections
I20260924 05:05:33.249265 392 mini_postgres.cc:96] Postgres bound to 56127
2026-09-24 05:05:33.254 UTC [12165] FATAL: database "slave" does not exist
127.0.98.20:56127 - accepting connections
I20260924 05:05:33.255260 392 mini_ranger.cc:161] Starting Ranger...
I20260924 05:05:33.268805 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 05:05:33.644537 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 05:05:33.644609 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-admin
2026-09-24 05:05:33,911 [I] DB FLAVOR :POSTGRES
2026-09-24 05:05:33,911 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:05:33,911 [I] Checking connection..
2026-09-24 05:05:33,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 05:05:34,147 [I] Checking connection passed.
2026-09-24 05:05:34,148 [I] --------- Verifying version history table ---------
2026-09-24 05:05:34,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:05:34,400 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:05:34,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:05:34,631 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:05:34,631 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 05:05:34,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 05:05:34.865 UTC [12229] WARNING: there is no transaction in progress
2026-09-24 05:05:34,871 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 05:05:34,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:05:35,130 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 05:05:35,130 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:05:35,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 05:05:35,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 05:05:35,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m01t.c.gcp-upstream.internal','N') ;"
2026-09-24 05:05:35,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:05:36,117 [I] Table x_portal_user does not exist in database ranger
2026-09-24 05:05:36,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:05:36,374 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 05:05:36,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:05:36,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:05:36,850 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 05:05:36,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 05:05:37.743 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.758 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.773 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.788 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.802 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.817 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.869 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.875 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.888 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.898 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.907 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.920 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.929 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.935 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.941 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:37.948 UTC [12337] WARNING: there is no transaction in progress
2026-09-24 05:05:38,494 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 05:05:38,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 05:05:38,745 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 05:05:38,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:05:39,003 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 05:05:39,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:05:39,241 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 05:05:39,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:05:39,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:05:39,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m01t.c.gcp-upstream.internal';"
2026-09-24 05:05:39,990 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 05:05:39,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 05:05:40,243 [I] DB_PATCHES have already been applied
I20260924 05:05:40.249316 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyORENn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 05:05:40.249392 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 05:05:40.252276 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52809
Sep 24, 2026 5:05:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:05:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:05:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:05:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:05:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:05:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:05:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 05:05:42.272785 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.018s user 0.000s sys 0.009s
Sep 24, 2026 5:06:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:06:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 5:06:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:06:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:06:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 05:06:02.243791 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 05:06:02 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 05:06:02.318053 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 05:06:02.326946 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 05:06:03.034113 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 05:06:03.034188 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms
I20260924 05:06:03.034204 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56127
2026-09-24 05:06:03,181 [I] DB FLAVOR :POSTGRES
2026-09-24 05:06:03,181 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:06:03,181 [I] Checking connection
2026-09-24 05:06:03,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 05:06:03,429 [I] connection success
2026-09-24 05:06:03,429 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 05:06:03,429 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 05:06:03,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 05:06:03,660 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 05:06:03,661 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:06:03,661 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 05:06:03,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56127/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 05:06:03,923 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 05:06:04.349665 392 mini_ranger_kms.cc:310] Created kms service
I20260924 05:06:04.584300 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 05:06:04.628682 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 05:06:04 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:05.026244 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 05:06:05.026358 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyORENn/build/release/bin/postgresql.jar:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyORENn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 05:06:05.026376 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 05:06:05.029543 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12518
I20260924 05:06:05.029627 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56629
Sep 24, 2026 5:06:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:06:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:06:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:06:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:06:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:06:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 05:06:07.076751 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.759s user 0.011s sys 0.119s
I20260924 05:06:07.076843 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 05:06:07.076879 392 mini_ranger_kms.cc:401] 127.0.98.20:56629/kms/v1/keys
Sep 24 05:06:07 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226367, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 05:06:07 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226367, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:06:08 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226367, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:06:10 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:06:10.361590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43639
--ranger_config_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:10.443200 12569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:10.443413 12569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:10.443434 12569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:10.444576 12569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:10.444628 12569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:10.444643 12569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:10.444654 12569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:10.444664 12569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:10.446331 12569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56629/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43639
--ranger_config_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:10.446560 12569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:10.446770 12569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:10.451435 12575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:10.451447 12574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:10.451589 12569 server_base.cc:1034] running on GCE node
W20260924 05:06:10.451687 12577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:10.451905 12569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:10.452191 12569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:10.453341 12569 hybrid_clock.cc:648] HybridClock initialized: now 1790226370453315 us; error 42 us; skew 500 ppm
Sep 24 05:06:10 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:10.459081 12569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:10.459597 12569 webserver.cc:460] Webserver started at http://127.0.98.62:45749/ using document root <none> and password file <none>
I20260924 05:06:10.459830 12569 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:10.459872 12569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:10.460005 12569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:06:10 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:10.515435 12569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "0061b960ab19444b9e703abbecb68f96"
format_stamp: "Formatted at 2026-09-24 05:06:10 on dist-test-slave-m01t"
server_key: "b60b7fdd44ea44e08d8761c83cd35330"
server_key_iv: "4eebba5f7abaf55f46f3c5613d009df9"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:10.515873 12569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "0061b960ab19444b9e703abbecb68f96"
format_stamp: "Formatted at 2026-09-24 05:06:10 on dist-test-slave-m01t"
server_key: "b60b7fdd44ea44e08d8761c83cd35330"
server_key_iv: "4eebba5f7abaf55f46f3c5613d009df9"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:10.517632 12569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:10.530200 12584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:10.530413 12569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:10.530521 12569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "0061b960ab19444b9e703abbecb68f96"
format_stamp: "Formatted at 2026-09-24 05:06:10 on dist-test-slave-m01t"
server_key: "b60b7fdd44ea44e08d8761c83cd35330"
server_key_iv: "4eebba5f7abaf55f46f3c5613d009df9"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:10.530617 12569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:10.544051 12569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:10.546025 12569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:10.546164 12569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:10.550629 12569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43639
I20260924 05:06:10.550657 12636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43639 every 8 connection(s)
I20260924 05:06:10.551025 12569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:10.551793 12637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:10.555003 12637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Bootstrap starting.
I20260924 05:06:10.555750 12637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:10.556080 12637 log.cc:824] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:10.556864 12637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: No bootstrap required, opened a new log
I20260924 05:06:10.557343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 12569
I20260924 05:06:10.557425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:06:10.557588 392 external_mini_cluster.cc:1428] Setting key 9c2155f76ec06ecaa7ad4be216f9791a
I20260924 05:06:10.559764 12637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:10.559995 12637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:10.560047 12637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061b960ab19444b9e703abbecb68f96, State: Initialized, Role: FOLLOWER
I20260924 05:06:10.560178 12637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:10.560277 12637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:10.560340 12637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:10.560402 12637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:10.561148 12637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:10.561261 12637 leader_election.cc:302] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0061b960ab19444b9e703abbecb68f96; no voters:
I20260924 05:06:10.561439 12637 leader_election.cc:288] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:06:10.561481 12642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:10.561750 12642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 LEADER]: Becoming Leader. State: Replica: 0061b960ab19444b9e703abbecb68f96, State: Running, Role: LEADER
I20260924 05:06:10.561851 12637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:10.561885 12642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:10.562238 12642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0061b960ab19444b9e703abbecb68f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260924 05:06:10.562300 12642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:10.562450 12644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0061b960ab19444b9e703abbecb68f96. Latest consensus state: current_term: 1 leader_uuid: "0061b960ab19444b9e703abbecb68f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260924 05:06:10.562649 12644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:10.563781 12637 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:06:10.563969 12646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:10.564458 12646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:06:10 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:10.570371 12646 catalog_manager.cc:1347] Generated new cluster ID: 3678df3d783e40139cd2a893e421aa9e
I20260924 05:06:10.570633 12646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:10.572652 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:10.558060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.98.62:43639)
0924 05:06:10.558234 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:10.558243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:10.558273 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:06:10.559703 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:10.559710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:10.559761 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:10.559894 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:10.560678 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:10.561817 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:10.562887 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:10.563405 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:10.563795 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:10.566451 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:10.566497 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:10.566497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:10.566623 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:10.569771 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:10.570217 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:10.570221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:10.570221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:10.570360 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:10.570592 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:10.570593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:10.570593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:10.571023 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:10.571636 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:10.571710 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:06:10.572037 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260924 05:06:10.579192 12646 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:06:10.579943 12646 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:10.588346 12646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Generated new TSK 0
I20260924 05:06:10.588637 12646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:06:11.341614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:11.441690 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:11.441946 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:11.441980 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:11.443261 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:11.443316 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:11.443351 12682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:11.444949 12682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:11.445192 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:11.445436 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:11.450480 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:11.450503 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:11.450481 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:11.451036 12682 server_base.cc:1034] running on GCE node
I20260924 05:06:11.451232 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:11.451472 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:11.452616 12682 hybrid_clock.cc:648] HybridClock initialized: now 1790226371452600 us; error 31 us; skew 500 ppm
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:11.455658 12682 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:11.456105 12682 webserver.cc:460] Webserver started at http://127.0.98.1:37579/ using document root <none> and password file <none>
I20260924 05:06:11.456336 12682 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:11.456384 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:11.456512 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:11.470028 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddfa97e360174bffb8dd71d4b36a99a2"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "91ee0a0caed43dee921de0c71f6e457f"
server_key_iv: "e5104877f6663b1f357c7dfbff14d7a2"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.470602 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddfa97e360174bffb8dd71d4b36a99a2"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "91ee0a0caed43dee921de0c71f6e457f"
server_key_iv: "e5104877f6663b1f357c7dfbff14d7a2"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.472478 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:11.495159 12699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.495430 12682 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260924 05:06:11.495554 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "ddfa97e360174bffb8dd71d4b36a99a2"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "91ee0a0caed43dee921de0c71f6e457f"
server_key_iv: "e5104877f6663b1f357c7dfbff14d7a2"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.495728 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:11.510848 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:11.513466 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:11.513638 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:11.513897 12682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:11.514333 12682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:11.514384 12682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.514413 12682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:11.514460 12682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.521018 12682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33713
I20260924 05:06:11.521584 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:11.523582 12812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33713 every 8 connection(s)
I20260924 05:06:11.528391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 12682
I20260924 05:06:11.528508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:06:11.528697 392 external_mini_cluster.cc:1428] Setting key bbc4202684fe17c4b837caed35446f55
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:11.536140 12815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:11.524351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.1:44101)
0924 05:06:11.524977 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:11.525027 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:11.525683 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:11.526176 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:11.526181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:11.526457 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:11.526758 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:11.526767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.527803 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.527805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:11.529245 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:11.529249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.529812 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.529813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:11.529960 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:11.530553 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:11.530563 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:11.532179 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:11.534370 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:11.534379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:11.534382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:11.534698 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:11.534926 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:11.534928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:11.534928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:11.535072 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:11.535396 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:11.535404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:11.535604 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 05:06:11.535714 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 05:06:11.535888 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":87,"threads_started":1}
I20260924 05:06:11.536625 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:11.524675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44101 (local address 127.0.98.62:43639)
0924 05:06:11.525924 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:11.525926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:11.525934 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:11.525961 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:11.525962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:11.525973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:11.526038 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:11.526873 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.527691 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.529357 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.529633 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.529695 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:11.532307 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:11.532309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:11.532310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:11.532318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:11.534279 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:11.534784 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:11.534786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:11.534787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:11.534860 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:11.535138 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:11.535139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:11.535139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:11.535306 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:11.535370 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:11.536433 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0924 05:06:11.536524 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":79,"threads_started":1}
I20260924 05:06:11.537125 12813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 05:06:11.537263 12813 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:11.537483 12813 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:11.538116 12601 ts_manager.cc:194] Registered new tserver with Master: ddfa97e360174bffb8dd71d4b36a99a2 (127.0.98.1:33713)
I20260924 05:06:11.539037 12601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44101
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:06:11.564013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:11.663878 12819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:11.664093 12819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:11.664127 12819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:11.665560 12819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:11.665616 12819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:11.665653 12819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:11.667385 12819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:11.667642 12819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:11.667876 12819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:11.672776 12829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:11.672758 12827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:11.673328 12819 server_base.cc:1034] running on GCE node
W20260924 05:06:11.673427 12826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:11.673633 12819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:11.673920 12819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:11.675068 12819 hybrid_clock.cc:648] HybridClock initialized: now 1790226371675056 us; error 27 us; skew 500 ppm
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:11.677424 12819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:11.677958 12819 webserver.cc:460] Webserver started at http://127.0.98.2:39907/ using document root <none> and password file <none>
I20260924 05:06:11.678187 12819 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:11.678234 12819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:11.678399 12819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:11.695147 12819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "8f87a9eff2be4f1fb184d57b1fd986e3"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "babbba7fea7a165ed7883236c345847b"
server_key_iv: "1451d9c9ed76c1eacb862899fd8eeff8"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.695729 12819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8f87a9eff2be4f1fb184d57b1fd986e3"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "babbba7fea7a165ed7883236c345847b"
server_key_iv: "1451d9c9ed76c1eacb862899fd8eeff8"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.709441 12819 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.000s
I20260924 05:06:11.741860 12836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.742126 12819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 05:06:11.742221 12819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "8f87a9eff2be4f1fb184d57b1fd986e3"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "babbba7fea7a165ed7883236c345847b"
server_key_iv: "1451d9c9ed76c1eacb862899fd8eeff8"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.742340 12819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:11.755808 12819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:11.758315 12819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:11.758520 12819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:11.758761 12819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:11.759171 12819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:11.759249 12819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.759320 12819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:11.759354 12819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.767874 12819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41431
I20260924 05:06:11.768458 12819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:06:11.768932 12950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41431 every 8 connection(s)
I20260924 05:06:11.771827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 12819
I20260924 05:06:11.771930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:06:11.772100 392 external_mini_cluster.cc:1428] Setting key 90919055c0503c74fda2181ce96fae51
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:11.782955 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:11.770101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53985 (local address 127.0.98.62:43639)
0924 05:06:11.770149 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:11.770151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:11.771240 (+ 1089us) server_negotiation.cc:407] Connection header received
0924 05:06:11.772013 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:11.772015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:11.772028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:11.772058 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:11.772980 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.773844 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.775458 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.775771 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.775819 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:11.778531 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:11.778533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:11.778534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:11.778542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:11.780271 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:11.780778 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:11.780779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:11.780780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:11.780841 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:11.781120 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:11.781121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:11.781121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:11.781275 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:11.781322 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:11.781788 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 05:06:11.781886 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:06:11.783794 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:11.769801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.2:53985)
0924 05:06:11.771079 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:11.771130 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:11.771884 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:11.772246 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:11.772251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:11.772574 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:11.772867 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:11.772876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.773956 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.773956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:11.775360 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:11.775363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:11.775930 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:11.775931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:11.776076 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:11.776690 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:11.776701 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:11.778414 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:11.780364 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:11.780373 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:11.780376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:11.780688 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:11.780910 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:11.780911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:11.780911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:11.781053 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:11.781371 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:11.781378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:11.781584 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 05:06:11.783341 (+ 1757us) client_negotiation.cc:240] Negotiation successful
0924 05:06:11.783537 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"thread_start_us":89,"threads_started":1}
I20260924 05:06:11.784296 12951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 05:06:11.784396 12951 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:11.784593 12951 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:11.785015 12601 ts_manager.cc:194] Registered new tserver with Master: 8f87a9eff2be4f1fb184d57b1fd986e3 (127.0.98.2:41431)
I20260924 05:06:11.785516 12601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53985
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:06:11.812723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:11.937811 12956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:11.938174 12956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:11.938256 12956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:11.940092 12956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:11.940248 12956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:11.940380 12956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:11.942876 12956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:11.943346 12956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:11.943675 12956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:11.950004 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:11.950443 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:11.951157 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:11.951400 12956 server_base.cc:1034] running on GCE node
I20260924 05:06:11.951762 12956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:11.952064 12956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:11.953342 12956 hybrid_clock.cc:648] HybridClock initialized: now 1790226371953323 us; error 32 us; skew 500 ppm
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:11.956681 12956 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:11.957191 12956 webserver.cc:460] Webserver started at http://127.0.98.3:45067/ using document root <none> and password file <none>
I20260924 05:06:11.957427 12956 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:11.957471 12956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:11.957588 12956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:06:11 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:11.970829 12956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f1f09efe1a74f478abadc7ca33996f0"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "ea69d57c08fcc8cfdf0b14debd7cd7d0"
server_key_iv: "c04c08a219241abd894bc5110b55ebed"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.971273 12956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f1f09efe1a74f478abadc7ca33996f0"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "ea69d57c08fcc8cfdf0b14debd7cd7d0"
server_key_iv: "c04c08a219241abd894bc5110b55ebed"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.973039 12956 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 05:06:11.986657 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:11.986887 12956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:11.986982 12956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "2f1f09efe1a74f478abadc7ca33996f0"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "ea69d57c08fcc8cfdf0b14debd7cd7d0"
server_key_iv: "c04c08a219241abd894bc5110b55ebed"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:11.987079 12956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:12.012364 12956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:12.014809 12956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:12.014953 12956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:12.015187 12956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:12.015522 12956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:12.015565 12956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.015597 12956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:12.015612 12956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.021579 12956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42583
I20260924 05:06:12.022044 12956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:06:12.023885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 12956
I20260924 05:06:12.023988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:06:12.024173 392 external_mini_cluster.cc:1428] Setting key c043ff5622d6e2e5f5213ef49756fdfa
I20260924 05:06:12.026876 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42583 every 8 connection(s)
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:12.039217 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.027535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60791 (local address 127.0.98.62:43639)
0924 05:06:12.027577 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:12.027579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:12.028075 (+ 496us) server_negotiation.cc:407] Connection header received
0924 05:06:12.028741 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:12.028742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:12.028755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:12.028782 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:12.029625 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.030462 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.031966 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.032270 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.032321 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.035157 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:12.035158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:12.035158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:12.035167 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:12.037468 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.038119 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.038120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.038121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.038181 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.038472 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.038473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.038474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.038607 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:12.038644 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:12.039029 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 05:06:12.039122 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:06:12.040117 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.027682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.3:60791)
0924 05:06:12.027957 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:12.027999 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:12.028626 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:12.028980 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:12.028984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:12.029226 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:12.029521 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.029528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.030579 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.030580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.031872 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.031875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.032423 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.032424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.032566 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.033250 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:12.033261 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:12.035042 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:12.037688 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.037698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.037701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.038035 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.038245 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.038245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.038245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.038407 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.038684 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:12.038689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:12.038884 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 05:06:12.039705 (+ 821us) client_negotiation.cc:240] Negotiation successful
0924 05:06:12.039879 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260924 05:06:12.040788 13085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 05:06:12.040882 13085 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.041067 13085 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:12.041465 12601 ts_manager.cc:194] Registered new tserver with Master: 2f1f09efe1a74f478abadc7ca33996f0 (127.0.98.3:42583)
I20260924 05:06:12.042003 12601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60791
I20260924 05:06:12.046892 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:06:12.047646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 12682
I20260924 05:06:12.054262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 12819
I20260924 05:06:12.060313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 12956
I20260924 05:06:12.066773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 12569
I20260924 05:06:12.071115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--webserver_interface=127.0.98.62
--webserver_port=45749
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43639
--ranger_config_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:12.153554 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:12.153832 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:12.153867 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:12.155088 13088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:12.155136 13088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:12.155154 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:12.155170 13088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:12.155187 13088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:12.156944 13088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43639
--ranger_config_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:12.157194 13088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:12.157414 13088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:12.161948 13094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.161957 13096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.161960 13093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:12.162302 13088 server_base.cc:1034] running on GCE node
I20260924 05:06:12.162525 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:12.162804 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:12.163959 13088 hybrid_clock.cc:648] HybridClock initialized: now 1790226372163940 us; error 34 us; skew 500 ppm
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:12.166358 13088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:12.166822 13088 webserver.cc:460] Webserver started at http://127.0.98.62:45749/ using document root <none> and password file <none>
I20260924 05:06:12.167045 13088 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:12.167102 13088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:12.229794 13088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:12.230494 13104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.230660 13088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:12.230751 13088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "0061b960ab19444b9e703abbecb68f96"
format_stamp: "Formatted at 2026-09-24 05:06:10 on dist-test-slave-m01t"
server_key: "b60b7fdd44ea44e08d8761c83cd35330"
server_key_iv: "4eebba5f7abaf55f46f3c5613d009df9"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:12.231117 13088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:12.251057 13088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:12.252691 13088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:12.252840 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:12.256825 13088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43639
I20260924 05:06:12.256892 13156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43639 every 8 connection(s)
I20260924 05:06:12.257236 13088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:12.257493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13088
I20260924 05:06:12.257798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33713
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43639
--webserver_port=37579
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:12.258575 13157 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:06:12.259209 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Bootstrap starting.
I20260924 05:06:12.266811 13157 log.cc:824] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:12.268504 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:12.268754 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Bootstrap complete.
I20260924 05:06:12.271062 13157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:12.271312 13157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061b960ab19444b9e703abbecb68f96, State: Initialized, Role: FOLLOWER
I20260924 05:06:12.271467 13157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:12.271538 13157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:12.271586 13157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:12.271632 13157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:06:12.272629 13157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:12.272725 13157 leader_election.cc:302] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0061b960ab19444b9e703abbecb68f96; no voters:
I20260924 05:06:12.272890 13157 leader_election.cc:288] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:06:12.272943 13161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:06:12.273046 13161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [term 2 LEADER]: Becoming Leader. State: Replica: 0061b960ab19444b9e703abbecb68f96, State: Running, Role: LEADER
I20260924 05:06:12.273128 13161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260924 05:06:12.273175 13157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:12.273381 13161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0061b960ab19444b9e703abbecb68f96. Latest consensus state: current_term: 2 leader_uuid: "0061b960ab19444b9e703abbecb68f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260924 05:06:12.273444 13161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:12.273571 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0061b960ab19444b9e703abbecb68f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0061b960ab19444b9e703abbecb68f96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260924 05:06:12.273628 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:12.273949 13165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:12.274441 13165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:12.274559 13165 catalog_manager.cc:1259] Loaded cluster ID: 3678df3d783e40139cd2a893e421aa9e
I20260924 05:06:12.274592 13165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:12.274915 13157 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:06:12.275710 13165 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:12.275895 13165 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Loaded TSK: 0
I20260924 05:06:12.276110 13165 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:06:12.361981 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:12.362206 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:12.362252 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:12.363484 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:12.363528 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:12.363564 13159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:12.365307 13159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37579
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:12.365557 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:12.365809 13159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:12.370870 13197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.371025 13194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.371137 13195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:12.371743 13159 server_base.cc:1034] running on GCE node
I20260924 05:06:12.371942 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:12.372169 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:12.373312 13159 hybrid_clock.cc:648] HybridClock initialized: now 1790226372373297 us; error 24 us; skew 500 ppm
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:12.375846 13159 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:12.376310 13159 webserver.cc:460] Webserver started at http://127.0.98.1:37579/ using document root <none> and password file <none>
I20260924 05:06:12.376536 13159 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:12.376600 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:12.402598 13159 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:12.404371 13205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.404531 13159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:12.404618 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "ddfa97e360174bffb8dd71d4b36a99a2"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "91ee0a0caed43dee921de0c71f6e457f"
server_key_iv: "e5104877f6663b1f357c7dfbff14d7a2"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:12.405059 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:12.422914 13159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:12.425108 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:12.425253 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:12.425488 13159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:12.425855 13159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:12.425904 13159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.425932 13159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:12.425953 13159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.432727 13159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33713
I20260924 05:06:12.432811 13318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33713 every 8 connection(s)
I20260924 05:06:12.433144 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:12.435060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13159
I20260924 05:06:12.435364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41431
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43639
--webserver_port=39907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:12.448460 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.434216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38453 (local address 127.0.98.62:43639)
0924 05:06:12.437885 (+ 3669us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:12.437898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:12.437948 (+ 50us) server_negotiation.cc:407] Connection header received
0924 05:06:12.438127 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:12.438136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:12.438198 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:12.438356 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:12.438981 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.439973 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.441528 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.442214 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.442815 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.444541 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:12.444546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:12.444547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:12.444559 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:12.446799 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.447189 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.447192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.447193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.447282 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.447513 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.447514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.447514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.447748 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:12.447814 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:12.448119 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 05:06:12.448231 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3588,"thread_start_us":109,"threads_started":1}
I20260924 05:06:12.448472 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.433917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.1:38453)
0924 05:06:12.434080 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:12.434134 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:12.434776 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:12.438415 (+ 3639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:12.438420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:12.438645 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:12.438853 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.438858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.440104 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.440105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.441400 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.441404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.442342 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.442343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.442492 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.443128 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:12.443138 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:12.444435 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:12.446885 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.446892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.446895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.447115 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.447355 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.447356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.447356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.447461 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.447823 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:12.447829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:12.447972 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 05:06:12.448061 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 05:06:12.448231 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260924 05:06:12.449577 13319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 05:06:12.449672 13319 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.449911 13319 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
W20260924 05:06:12.450716 13319 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:06:12.450785 13319 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.450850 13319 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:12.451169 13319 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.451244 13319 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
W20260924 05:06:12.451514 13319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:06:12.542101 13323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:12.542397 13323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:12.542443 13323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:12.544227 13323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:12.544298 13323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:12.544338 13323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:12.546598 13323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39907
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:12.546864 13323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:12.547093 13323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:12.551920 13328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.551915 13329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.552100 13331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:12.552918 13323 server_base.cc:1034] running on GCE node
I20260924 05:06:12.553117 13323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:12.553388 13323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:12.554554 13323 hybrid_clock.cc:648] HybridClock initialized: now 1790226372554536 us; error 35 us; skew 500 ppm
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:12.557090 13323 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:12.557538 13323 webserver.cc:460] Webserver started at http://127.0.98.2:39907/ using document root <none> and password file <none>
I20260924 05:06:12.557782 13323 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:12.557865 13323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:12.577780 13323 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260924 05:06:12.580945 13339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.581142 13323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 05:06:12.581249 13323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "8f87a9eff2be4f1fb184d57b1fd986e3"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "babbba7fea7a165ed7883236c345847b"
server_key_iv: "1451d9c9ed76c1eacb862899fd8eeff8"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:12.581804 13323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:12.621073 13323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:12.623224 13323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:12.623373 13323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:12.623626 13323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:12.624002 13323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:12.624042 13323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.624070 13323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:12.624091 13323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.630677 13323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41431
I20260924 05:06:12.631060 13323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:06:12.631213 13452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41431 every 8 connection(s)
I20260924 05:06:12.632167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13323
I20260924 05:06:12.632449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42583
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43639
--webserver_port=45067
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:12.641289 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.632030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49749 (local address 127.0.98.62:43639)
0924 05:06:12.632087 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:12.632089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:12.632397 (+ 308us) server_negotiation.cc:407] Connection header received
0924 05:06:12.633028 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:12.633029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:12.633046 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:12.633121 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:12.633907 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.634515 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.635820 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.636029 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.636088 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.638231 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:12.638232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:12.638233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:12.638241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:12.639524 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.640298 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.640300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.640300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.640358 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.640613 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.640614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.640615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.640737 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:12.640796 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:12.641096 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 05:06:12.641192 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:06:12.641698 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.632025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.2:49749)
0924 05:06:12.632189 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:12.632282 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:12.632901 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:12.633236 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:12.633241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:12.633466 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:12.633789 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.633796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.634609 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.634610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.635735 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.635738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.636169 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.636170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.636328 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.636839 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:12.636848 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:12.638135 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:12.639602 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.639609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.639611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.639850 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.640432 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.640433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.640433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.640557 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.640795 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:12.640802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:12.640971 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 05:06:12.641068 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 05:06:12.641239 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260924 05:06:12.642206 13453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 05:06:12.642306 13453 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.642493 13453 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
W20260924 05:06:12.643106 13453 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:06:12.643173 13453 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.643240 13453 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:12.643522 13453 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.643595 13453 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
W20260924 05:06:12.644384 13453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:06:12.753509 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:12.753861 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:12.753907 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:12.755726 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:12.755791 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:12.755833 13456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:12.758112 13456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45067
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:12.758397 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:12.758665 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:12.767478 13461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:12.767589 13456 server_base.cc:1034] running on GCE node
W20260924 05:06:12.767712 13464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:12.767871 13462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:12.768185 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:12.768501 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:12.769687 13456 hybrid_clock.cc:648] HybridClock initialized: now 1790226372769667 us; error 33 us; skew 500 ppm
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:12.772740 13456 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:12.773232 13456 webserver.cc:460] Webserver started at http://127.0.98.3:45067/ using document root <none> and password file <none>
I20260924 05:06:12.773454 13456 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:12.773516 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:12.791735 13456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:12.792478 13473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.792709 13456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:12.792814 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "2f1f09efe1a74f478abadc7ca33996f0"
format_stamp: "Formatted at 2026-09-24 05:06:11 on dist-test-slave-m01t"
server_key: "ea69d57c08fcc8cfdf0b14debd7cd7d0"
server_key_iv: "c04c08a219241abd894bc5110b55ebed"
server_key_version: "kuduclusterkey@0"
I20260924 05:06:12.793166 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:12.822309 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:12.824435 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:12.824602 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:12.824893 13456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:12.825299 13456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:12.825350 13456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.825389 13456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:12.825412 13456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:12.832235 13456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42583
I20260924 05:06:12.832448 13586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42583 every 8 connection(s)
I20260924 05:06:12.832679 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:06:12.839035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13456
Sep 24 05:06:12 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:12.843525 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.833391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33011 (local address 127.0.98.62:43639)
0924 05:06:12.833441 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:12.833443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:12.833745 (+ 302us) server_negotiation.cc:407] Connection header received
0924 05:06:12.834422 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:12.834423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:12.834434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:12.834513 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:12.835217 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.835699 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.837138 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.837337 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.837395 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.839976 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:12.839978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:12.839978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:12.839986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:12.841684 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.842300 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.842302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.842302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.842371 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.842688 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.842689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.842689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.842842 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:12.842889 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:12.843330 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 05:06:12.843425 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:06:12.844098 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.833420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.3:33011)
0924 05:06:12.833588 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:12.833637 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:12.834310 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:12.834595 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:12.834599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:12.834844 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:12.835132 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.835139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.835797 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.835798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.837054 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:12.837057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.837437 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.837438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:12.837579 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.838156 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:12.838167 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:12.839864 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:12.841806 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.841815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.841818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.842145 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.842449 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:12.842450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:12.842450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:12.842607 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:12.842937 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:12.842944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:12.843148 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 05:06:12.843666 (+ 518us) client_negotiation.cc:240] Negotiation successful
0924 05:06:12.843852 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260924 05:06:12.844589 13587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260924 05:06:12.844705 13587 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.844910 13587 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
W20260924 05:06:12.845584 13587 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:06:12.845646 13587 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.845695 13587 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:12.846020 13587 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:12.846100 13587 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
W20260924 05:06:12.846369 13587 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:06:12.854342 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:12.839640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.0.98.62:43639)
0924 05:06:12.842726 (+ 3086us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:12.842728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:12.842739 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:06:12.842762 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:12.842763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:12.842777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:12.843878 (+ 1101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:12.843917 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.844869 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.849818 (+ 4949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:12.850156 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:12.850259 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:12.851541 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:12.851543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:12.851543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:12.851552 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:12.853197 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.853532 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.853534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.853534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.853603 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:12.853782 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:12.853783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:12.853783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:12.853933 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:12.854113 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:12.854153 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 05:06:12.854238 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
Sep 24 05:06:13 dist-test-slave-m01t krb5kdc[12124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:13.451675 13319 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:13.451821 13319 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:13.452189 13121 ts_manager.cc:194] Registered new tserver with Master: ddfa97e360174bffb8dd71d4b36a99a2 (127.0.98.1:33713)
I20260924 05:06:13.452843 13121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38453
I20260924 05:06:13.644554 13453 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:13.644719 13453 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:13.645051 13121 ts_manager.cc:194] Registered new tserver with Master: 8f87a9eff2be4f1fb184d57b1fd986e3 (127.0.98.2:41431)
I20260924 05:06:13.645567 13121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49749
Sep 24 05:06:13 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:06:13.846544 13587 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:13.846689 13587 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260924 05:06:13.847005 13121 ts_manager.cc:194] Registered new tserver with Master: 2f1f09efe1a74f478abadc7ca33996f0 (127.0.98.3:42583)
I20260924 05:06:13.847481 13121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33011
I20260924 05:06:13.848670 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:06:13.857681 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.849944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.0.98.62:43639)
0924 05:06:13.850194 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:13.850196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:13.850203 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:06:13.850224 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:13.850224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:13.850235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:13.850915 (+ 680us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:13.850955 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.851645 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.852624 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.852923 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.853253 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.854392 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:13.854393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:13.854393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:13.854400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:13.856031 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:13.856401 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:13.856403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:13.856403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:13.856461 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:13.856668 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:13.856669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:13.856669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:13.856827 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:13.857409 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:13.857463 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 05:06:13.857567 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":67,"threads_started":1}
I20260924 05:06:13.860175 13121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:06:13.862970 13121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:06:13.877549 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.872435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41431 (local address 127.0.0.1:53548)
0924 05:06:13.873183 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:13.873226 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:13.873362 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:13.873674 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:13.873676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:13.873679 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:13.873812 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:13.873815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.875087 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.875088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:13.876304 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:13.876307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.877246 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.877247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:13.877336 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.877351 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:06:13.877416 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 05:06:13.877464 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":102,"threads_started":1}
I20260924 05:06:13.877887 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.872855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.98.1:33713)
0924 05:06:13.873280 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:13.873284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:13.873513 (+ 229us) server_negotiation.cc:407] Connection header received
0924 05:06:13.873548 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:13.873550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:13.873564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:13.873628 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:13.873894 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.875197 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.876413 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.877407 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.877460 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.877530 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:13.877773 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 05:06:13.877814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":58,"threads_started":1}
I20260924 05:06:13.878185 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.872716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.2:41431)
0924 05:06:13.873346 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:13.873350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:13.873360 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:06:13.873512 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:13.873515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:13.873532 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:13.873592 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:13.873885 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.874999 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.876394 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.877094 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.877157 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.877214 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:13.877904 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 05:06:13.878109 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":48,"threads_started":1}
I20260924 05:06:13.878505 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.872435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33713 (local address 127.0.0.1:40942)
0924 05:06:13.873408 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:13.873417 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:13.873467 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:13.873698 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:13.873699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:13.873700 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:13.873817 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:13.873819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.875268 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.875269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:13.876324 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:13.876327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.877560 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.877561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:13.877618 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.877622 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:06:13.877661 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 05:06:13.877698 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260924 05:06:13.879232 13253 tablet_service.cc:1467] Processing CreateTablet for tablet 047985f4aede45abb1dada25b7e65ff5 (DEFAULT_TABLE table=test-table [id=3791d353408a410684e40e8c53e39c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:13.879236 13387 tablet_service.cc:1467] Processing CreateTablet for tablet 047985f4aede45abb1dada25b7e65ff5 (DEFAULT_TABLE table=test-table [id=3791d353408a410684e40e8c53e39c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:13.879580 13387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047985f4aede45abb1dada25b7e65ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:13.879580 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047985f4aede45abb1dada25b7e65ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:13.882345 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.873002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.0.98.3:42583)
0924 05:06:13.873220 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:13.873225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:13.878093 (+ 4868us) server_negotiation.cc:407] Connection header received
0924 05:06:13.878196 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:13.878199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:13.878215 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:13.878270 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:13.878642 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.879640 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.881191 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.881905 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.881951 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.882021 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:13.882225 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 05:06:13.882269 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":53,"threads_started":1}
I20260924 05:06:13.882544 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:13.872641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42583 (local address 127.0.0.1:55550)
0924 05:06:13.878021 (+ 5380us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:13.878032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:13.878122 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:13.878363 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:13.878364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:13.878365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:13.878401 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:13.878402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.879736 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.879746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:13.881094 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:13.881098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:13.882077 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:13.882078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:13.882138 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:13.882141 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:06:13.882430 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 05:06:13.882476 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5364,"thread_start_us":52,"threads_started":1}
I20260924 05:06:13.883389 13521 tablet_service.cc:1467] Processing CreateTablet for tablet 047985f4aede45abb1dada25b7e65ff5 (DEFAULT_TABLE table=test-table [id=3791d353408a410684e40e8c53e39c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:13.883709 13521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047985f4aede45abb1dada25b7e65ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:13.883749 13621 tablet_bootstrap.cc:492] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Bootstrap starting.
I20260924 05:06:13.884368 13621 tablet_bootstrap.cc:654] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:13.884634 13621 log.cc:824] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:13.885320 13621 tablet_bootstrap.cc:492] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: No bootstrap required, opened a new log
I20260924 05:06:13.885375 13621 ts_tablet_manager.cc:1397] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:13.888386 13621 raft_consensus.cc:348] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:13.889014 13621 raft_consensus.cc:374] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:13.889132 13621 raft_consensus.cc:729] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f87a9eff2be4f1fb184d57b1fd986e3, State: Initialized, Role: FOLLOWER
I20260924 05:06:13.889386 13621 consensus_queue.cc:260] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:13.889812 13621 ts_tablet_manager.cc:1428] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 05:06:13.889976 13453 heartbeater.cc:503] Master 127.0.98.62:43639 was elected leader, sending a full tablet report...
I20260924 05:06:13.890450 13623 tablet_bootstrap.cc:492] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Bootstrap starting.
I20260924 05:06:13.891208 13623 tablet_bootstrap.cc:654] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:13.891597 13623 log.cc:824] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:13.893936 13620 tablet_bootstrap.cc:492] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: Bootstrap starting.
I20260924 05:06:13.894737 13620 tablet_bootstrap.cc:654] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:13.895108 13620 log.cc:824] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:13.896025 13620 tablet_bootstrap.cc:492] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: No bootstrap required, opened a new log
I20260924 05:06:13.896095 13620 ts_tablet_manager.cc:1397] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:06:13.898813 13620 raft_consensus.cc:348] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:13.898991 13620 raft_consensus.cc:374] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:13.899024 13620 raft_consensus.cc:729] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa97e360174bffb8dd71d4b36a99a2, State: Initialized, Role: FOLLOWER
I20260924 05:06:13.899350 13620 consensus_queue.cc:260] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:13.899711 13620 ts_tablet_manager.cc:1428] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 05:06:13.899871 13319 heartbeater.cc:503] Master 127.0.98.62:43639 was elected leader, sending a full tablet report...
I20260924 05:06:13.902451 13623 tablet_bootstrap.cc:492] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: No bootstrap required, opened a new log
I20260924 05:06:13.902534 13623 ts_tablet_manager.cc:1397] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260924 05:06:13.905042 13623 raft_consensus.cc:348] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:13.905210 13623 raft_consensus.cc:374] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:13.905236 13623 raft_consensus.cc:729] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f09efe1a74f478abadc7ca33996f0, State: Initialized, Role: FOLLOWER
I20260924 05:06:13.905366 13623 consensus_queue.cc:260] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:13.905620 13623 ts_tablet_manager.cc:1428] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:06:13.905769 13587 heartbeater.cc:503] Master 127.0.98.62:43639 was elected leader, sending a full tablet report...
W20260924 05:06:13.934657 13320 tablet.cc:2369] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 05:06:14 dist-test-slave-m01t krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 05:06:14.084249 13588 tablet.cc:2369] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:06:14.132458 13454 tablet.cc:2369] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:06:14.169219 13628 raft_consensus.cc:482] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:14.169368 13628 raft_consensus.cc:504] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:14.169807 13628 leader_election.cc:288] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f87a9eff2be4f1fb184d57b1fd986e3 (127.0.98.2:41431), ddfa97e360174bffb8dd71d4b36a99a2 (127.0.98.1:33713)
I20260924 05:06:14.174069 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:14.170248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47237 (local address 127.0.98.2:41431)
0924 05:06:14.170291 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:14.170293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:14.170395 (+ 102us) server_negotiation.cc:407] Connection header received
0924 05:06:14.170495 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:14.170496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:14.170507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:14.170534 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:14.170763 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.171479 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.172931 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.173541 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.173586 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:14.173597 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:14.173933 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 05:06:14.173997 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:06:14.174084 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:14.169984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41431 (local address 127.0.98.3:47237)
0924 05:06:14.170325 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:14.170335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:14.170424 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:14.170596 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:14.170597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:14.170598 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:14.170684 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:14.170686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.171580 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.171581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:14.172825 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:14.172829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.173755 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.173756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:14.173829 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:14.173834 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:06:14.173907 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 05:06:14.173952 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":88,"threads_started":1}
I20260924 05:06:14.174688 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047985f4aede45abb1dada25b7e65ff5" candidate_uuid: "2f1f09efe1a74f478abadc7ca33996f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" is_pre_election: true
I20260924 05:06:14.174860 13403 raft_consensus.cc:2454] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f09efe1a74f478abadc7ca33996f0 in term 0.
I20260924 05:06:14.175149 13477 leader_election.cc:302] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f09efe1a74f478abadc7ca33996f0, 8f87a9eff2be4f1fb184d57b1fd986e3; no voters:
I20260924 05:06:14.175287 13628 raft_consensus.cc:2781] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:06:14.175331 13628 raft_consensus.cc:482] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:06:14.175350 13628 raft_consensus.cc:3037] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:14.176283 13628 raft_consensus.cc:504] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:14.176424 13628 leader_election.cc:288] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [CANDIDATE]: Term 1 election: Requested vote from peers 8f87a9eff2be4f1fb184d57b1fd986e3 (127.0.98.2:41431), ddfa97e360174bffb8dd71d4b36a99a2 (127.0.98.1:33713)
I20260924 05:06:14.176855 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047985f4aede45abb1dada25b7e65ff5" candidate_uuid: "2f1f09efe1a74f478abadc7ca33996f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3"
I20260924 05:06:14.176949 13403 raft_consensus.cc:3037] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:14.177359 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:14.170216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36545 (local address 127.0.98.1:33713)
0924 05:06:14.170257 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:14.170259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:14.174220 (+ 3961us) server_negotiation.cc:407] Connection header received
0924 05:06:14.174318 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:14.174320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:14.174332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:14.174360 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:14.174501 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.175233 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.176478 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.177033 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.177072 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:14.177082 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:14.177255 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 05:06:14.177295 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:06:14.177553 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:14.170025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33713 (local address 127.0.98.3:36545)
0924 05:06:14.174168 (+ 4143us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:14.174179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:14.174250 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:14.174413 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:14.174413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:14.174414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:14.174442 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:14.174443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.175358 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.175359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:14.176379 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:14.176381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.177133 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.177133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:14.177187 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:14.177190 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:06:14.177458 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 05:06:14.177495 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"thread_start_us":92,"threads_started":1}
I20260924 05:06:14.177855 13403 raft_consensus.cc:2454] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1f09efe1a74f478abadc7ca33996f0 in term 1.
I20260924 05:06:14.178057 13273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047985f4aede45abb1dada25b7e65ff5" candidate_uuid: "2f1f09efe1a74f478abadc7ca33996f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" is_pre_election: true
I20260924 05:06:14.178210 13273 raft_consensus.cc:2454] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f09efe1a74f478abadc7ca33996f0 in term 0.
I20260924 05:06:14.178350 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047985f4aede45abb1dada25b7e65ff5" candidate_uuid: "2f1f09efe1a74f478abadc7ca33996f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa97e360174bffb8dd71d4b36a99a2"
I20260924 05:06:14.178437 13272 raft_consensus.cc:3037] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:14.179273 13272 raft_consensus.cc:2454] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1f09efe1a74f478abadc7ca33996f0 in term 1.
I20260924 05:06:14.179566 13474 leader_election.cc:302] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f09efe1a74f478abadc7ca33996f0, ddfa97e360174bffb8dd71d4b36a99a2; no voters:
I20260924 05:06:14.179662 13628 raft_consensus.cc:2781] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:14.179805 13628 raft_consensus.cc:686] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1f09efe1a74f478abadc7ca33996f0, State: Running, Role: LEADER
I20260924 05:06:14.179896 13628 consensus_queue.cc:237] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } }
I20260924 05:06:14.180751 13121 catalog_manager.cc:5495] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f1f09efe1a74f478abadc7ca33996f0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2f1f09efe1a74f478abadc7ca33996f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f09efe1a74f478abadc7ca33996f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42583 } health_report { overall_health: HEALTHY } } }
I20260924 05:06:14.231891 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:14.228664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.98.3:42583)
0924 05:06:14.228703 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:14.228705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:14.228712 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:06:14.228762 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:14.228763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:14.228786 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:14.228845 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:14.229007 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.229760 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.230805 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:14.231086 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:14.231137 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:14.231317 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:14.231452 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:06:14.231518 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:14.231769 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 05:06:14.231793 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:06:14.235411 13272 raft_consensus.cc:1261] T 047985f4aede45abb1dada25b7e65ff5 P ddfa97e360174bffb8dd71d4b36a99a2 [term 1 FOLLOWER]: Refusing update from remote peer 2f1f09efe1a74f478abadc7ca33996f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:14.235469 13403 raft_consensus.cc:1261] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 1 FOLLOWER]: Refusing update from remote peer 2f1f09efe1a74f478abadc7ca33996f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:14.235772 13628 consensus_queue.cc:1035] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa97e360174bffb8dd71d4b36a99a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:14.235958 13633 consensus_queue.cc:1035] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f87a9eff2be4f1fb184d57b1fd986e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:14.239843 13636 mvcc.cc:205] Tried to move back new op lower bound from 7332767228865118208 to 7332767228641366016. Current Snapshot: MvccSnapshot[applied={T|T < 7332767228865118208}]
I20260924 05:06:14.240511 13638 mvcc.cc:205] Tried to move back new op lower bound from 7332767228865118208 to 7332767228641366016. Current Snapshot: MvccSnapshot[applied={T|T < 7332767228865118208}]
I20260924 05:06:14.243012 13120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:14.243134 13120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:14.244434 13120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0061b960ab19444b9e703abbecb68f96: Sending DeleteTablet for 3 replicas of tablet 047985f4aede45abb1dada25b7e65ff5
I20260924 05:06:14.244899 13253 tablet_service.cc:1514] Processing DeleteTablet for tablet 047985f4aede45abb1dada25b7e65ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40942
I20260924 05:06:14.244997 13521 tablet_service.cc:1514] Processing DeleteTablet for tablet 047985f4aede45abb1dada25b7e65ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55550
I20260924 05:06:14.245054 13387 tablet_service.cc:1514] Processing DeleteTablet for tablet 047985f4aede45abb1dada25b7e65ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260924 05:06:14.245519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13159
I20260924 05:06:14.253859 13645 tablet_replica.cc:330] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: stopping tablet replica
I20260924 05:06:14.253844 13646 tablet_replica.cc:330] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: stopping tablet replica
I20260924 05:06:14.253999 13645 raft_consensus.cc:2229] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:06:14.254002 13646 raft_consensus.cc:2229] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:14.254105 13645 raft_consensus.cc:2258] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:14.254133 13646 raft_consensus.cc:2258] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:14.254662 13645 ts_tablet_manager.cc:1905] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:14.256377 13645 ts_tablet_manager.cc:1918] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:14.256474 13645 log.cc:1196] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/047985f4aede45abb1dada25b7e65ff5
I20260924 05:06:14.256721 13645 ts_tablet_manager.cc:1939] T 047985f4aede45abb1dada25b7e65ff5 P 2f1f09efe1a74f478abadc7ca33996f0: Deleting consensus metadata
I20260924 05:06:14.256729 13646 ts_tablet_manager.cc:1905] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:14.257241 13105 catalog_manager.cc:4841] TS 2f1f09efe1a74f478abadc7ca33996f0 (127.0.98.3:42583): tablet 047985f4aede45abb1dada25b7e65ff5 (table test-table [id=3791d353408a410684e40e8c53e39c97]) successfully deleted
I20260924 05:06:14.259435 13646 ts_tablet_manager.cc:1918] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:14.259512 13646 log.cc:1196] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/047985f4aede45abb1dada25b7e65ff5
I20260924 05:06:14.259758 13646 ts_tablet_manager.cc:1939] T 047985f4aede45abb1dada25b7e65ff5 P 8f87a9eff2be4f1fb184d57b1fd986e3: Deleting consensus metadata
I20260924 05:06:14.260188 13108 catalog_manager.cc:4841] TS 8f87a9eff2be4f1fb184d57b1fd986e3 (127.0.98.2:41431): tablet 047985f4aede45abb1dada25b7e65ff5 (table test-table [id=3791d353408a410684e40e8c53e39c97]) successfully deleted
W20260924 05:06:14.261560 13474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33713 (error 108)
W20260924 05:06:14.261874 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33713 (error 108)
I20260924 05:06:14.262027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13323
I20260924 05:06:14.269917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13456
I20260924 05:06:14.271337 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:14.270826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33713 (local address 127.0.0.1:40950)
0924 05:06:14.271236 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:14.271294 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
W20260924 05:06:14.271481 13105 catalog_manager.cc:4584] TS ddfa97e360174bffb8dd71d4b36a99a2 (127.0.98.1:33713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 047985f4aede45abb1dada25b7e65ff5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33713: connect: Connection refused (error 111)
I20260924 05:06:14.274806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13088
I20260924 05:06:14.279719 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 05:06:14.614075 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 05:06:14.614171 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 05:06:14.948271 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 05:06:14.948 UTC [12154] LOG: received smart shutdown request
2026-09-24 05:06:14.951 UTC [12154] LOG: background worker "logical replication launcher" (PID 12163) exited with exit code 1
2026-09-24 05:06:14.951 UTC [12158] LOG: shutting down
2026-09-24 05:06:15.091 UTC [12154] LOG: database system is shut down
2026-09-24T05:06:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46078 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:06:15 dist-test-slave-m01t krb5kdc[13660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:06:15 dist-test-slave-m01t krb5kdc[13660](info): set up 2 sockets
Sep 24 05:06:15 dist-test-slave-m01t krb5kdc[13660](info): commencing operation
krb5kdc: starting...
W20260924 05:06:16.125388 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:06:16.232501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38405,127.0.98.61:34471,127.0.98.60:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:16.312554 13673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:16.312788 13673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:16.312819 13673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:16.314004 13673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:16.314047 13673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:16.314065 13673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:16.314081 13673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:16.314096 13673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:16.315652 13673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38405,127.0.98.61:34471,127.0.98.60:45021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:16.315902 13673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:16.316110 13673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:16.320708 13681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:16.320713 13679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:16.320803 13673 server_base.cc:1034] running on GCE node
W20260924 05:06:16.320708 13678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:16.321276 13673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:16.321545 13673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:16.322685 13673 hybrid_clock.cc:648] HybridClock initialized: now 1790226376322669 us; error 32 us; skew 500 ppm
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:16.328364 13673 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:16.328846 13673 webserver.cc:460] Webserver started at http://127.0.98.62:46665/ using document root <none> and password file <none>
I20260924 05:06:16.329062 13673 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:16.329105 13673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:16.329205 13673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:16.330336 13673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "493c3d3d070b46ce905a0d3eb05ceb4a"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "43708a8aee20d1de55aff67729aeb8b4"
server_key_iv: "c34d5ad660b615a21deb679e86a068e1"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.330655 13673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "493c3d3d070b46ce905a0d3eb05ceb4a"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "43708a8aee20d1de55aff67729aeb8b4"
server_key_iv: "c34d5ad660b615a21deb679e86a068e1"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.331897 13673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:06:16.332573 13688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:16.332736 13673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:16.332810 13673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "493c3d3d070b46ce905a0d3eb05ceb4a"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "43708a8aee20d1de55aff67729aeb8b4"
server_key_iv: "c34d5ad660b615a21deb679e86a068e1"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.332867 13673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:16.346079 13673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:16.347826 13673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:16.347944 13673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:16.351677 13673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38405
I20260924 05:06:16.351727 13740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38405 every 8 connection(s)
I20260924 05:06:16.352059 13673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:16.352911 13741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:16.354782 13741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:06:16.356871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13673
I20260924 05:06:16.356987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:06:16.357189 392 external_mini_cluster.cc:1428] Setting key 695aa0a0c40afbf47f85dc5d0384929e
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:16.364691 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.355482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.98.62:38405)
0924 05:06:16.356221 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.356225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.356236 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:06:16.356308 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.356309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.356325 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.356389 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.357041 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.358254 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.359141 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.359586 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.359670 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.361746 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.361754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.361754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.361761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.363173 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.363612 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.363614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.363614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.363689 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.363914 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.363914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.363915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.364117 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.364173 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.364387 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.364500 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":74,"threads_started":1}
I20260924 05:06:16.364702 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.355460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38405 (local address 127.0.0.1:58446)
0924 05:06:16.355619 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.355653 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.356089 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.356426 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.356429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.356676 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.356920 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.356926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.358347 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.358348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.359060 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.359062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.359652 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.359652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.359703 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.360219 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.360228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.361634 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.363266 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.363274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.363277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.363536 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.363746 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.363747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.363747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.363856 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.364179 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.364185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.364267 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.364374 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.364579 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":47,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 05:06:16.365397 13741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:06:16.365764 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.365630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47816)
0924 05:06:16.365675 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.365738 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:06:16.365888 13691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
I20260924 05:06:16.366225 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.366136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47826)
0924 05:06:16.366182 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.366204 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 05:06:16.366307 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34471: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:06:16.378527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34471
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38405,127.0.98.61:34471,127.0.98.60:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:16.415555 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } attempt: 1
I20260924 05:06:16.416075 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.415926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47830)
0924 05:06:16.415990 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.416040 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:06:16.416405 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.416319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47834)
0924 05:06:16.416360 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.416387 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:06:16.416495 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34471: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
W20260924 05:06:16.457401 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:16.457608 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:16.457635 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:16.458783 13748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:16.458829 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:16.458842 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:16.458865 13748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:16.458887 13748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:16.460412 13748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38405,127.0.98.61:34471,127.0.98.60:45021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:16.460647 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:16.460860 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:16.465282 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:16.465287 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:16.465313 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:16.465412 13748 server_base.cc:1034] running on GCE node
I20260924 05:06:16.465703 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:16.466020 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:16.467150 13748 hybrid_clock.cc:648] HybridClock initialized: now 1790226376467141 us; error 26 us; skew 500 ppm
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:16.469483 13748 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:06:16.469936 13748 webserver.cc:460] Webserver started at http://127.0.98.61:36545/ using document root <none> and password file <none>
I20260924 05:06:16.470129 13748 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:16.470175 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:16.470299 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:16.471326 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data/instance:
uuid: "eac942bd5dd44acd8a5da70949118062"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "010d90a54bff595d819706d9a3a380a3"
server_key_iv: "05ef47724c96e8915f8a8c54873af24d"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.471657 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal/instance:
uuid: "eac942bd5dd44acd8a5da70949118062"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "010d90a54bff595d819706d9a3a380a3"
server_key_iv: "05ef47724c96e8915f8a8c54873af24d"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.472877 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:06:16.473493 13763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:16.473636 13748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:16.473738 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal
uuid: "eac942bd5dd44acd8a5da70949118062"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "010d90a54bff595d819706d9a3a380a3"
server_key_iv: "05ef47724c96e8915f8a8c54873af24d"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.473801 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:16.484658 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } attempt: 2
I20260924 05:06:16.485041 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.484931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47850)
0924 05:06:16.484982 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.485010 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:06:16.485363 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.485280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47854)
0924 05:06:16.485320 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.485344 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:06:16.485447 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34471: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34471: connect: Connection refused (error 111)
I20260924 05:06:16.485962 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:16.487658 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:16.487764 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:16.491533 13748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34471
I20260924 05:06:16.491581 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34471 every 8 connection(s)
I20260924 05:06:16.491885 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:06:16.492523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13748
I20260924 05:06:16.492630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-1/wal/instance
I20260924 05:06:16.492825 392 external_mini_cluster.cc:1428] Setting key 2b27ba8f61d57377abbd2cf38989aa89
I20260924 05:06:16.492847 13816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:16.494791 13816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:16.505287 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.495511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.0.98.62:38405)
0924 05:06:16.495562 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.495564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.495791 (+ 227us) server_negotiation.cc:407] Connection header received
0924 05:06:16.496272 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.496273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.496286 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.496358 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.496952 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.497731 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.498685 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.499002 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.499075 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.502708 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.502709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.502710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.502718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.503981 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.504404 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.504404 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.504405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.504450 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.504681 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.504681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.504681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.504778 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.504832 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.505143 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.505207 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:06:16.505525 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.495509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38405 (local address 127.0.0.1:58450)
0924 05:06:16.495667 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.495702 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.496176 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.496412 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.496416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.496664 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.496860 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.496864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.497819 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.497820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.498607 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.498608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.499072 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.499073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.499165 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.499629 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.499637 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.502619 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.504055 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.504062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.504064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.504339 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.504498 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.504499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.504499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.504629 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.504833 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.504839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.505017 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.505131 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.505297 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260924 05:06:16.506109 13816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:06:16.513120 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.506363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47862)
0924 05:06:16.506412 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.506424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.506466 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.506657 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.506658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.506790 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.506847 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.506849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.507652 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.507653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.508435 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.508438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.508837 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.508838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.508916 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.509228 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.509232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.510882 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.512344 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.512345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.512345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.512483 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.512624 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.512624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.512624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.512670 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.512863 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.512864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.512895 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.512970 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.513052 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:06:16.513208 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.506398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.0.98.61:34471)
0924 05:06:16.506542 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.506546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.506558 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:06:16.506576 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.506578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.506588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.506643 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.506916 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.507578 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.508503 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.508775 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.508820 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.510958 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.510965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.510966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.510971 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.512257 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.512536 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.512538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.512538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.512584 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.512706 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.512706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.512707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.512820 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.512874 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.512991 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.513108 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260924 05:06:16.513679 13816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:06:16.514006 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.513905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44168)
0924 05:06:16.513935 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.513985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 05:06:16.514123 13767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
I20260924 05:06:16.514479 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.514405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44180)
0924 05:06:16.514443 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.514465 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:06:16.514564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45021
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38405,127.0.98.61:34471,127.0.98.60:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:16.514571 13816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
I20260924 05:06:16.563810 13816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } attempt: 1
I20260924 05:06:16.564373 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.564205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44196)
0924 05:06:16.564280 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.564333 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 05:06:16.564739 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.564646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44200)
0924 05:06:16.564690 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.564718 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:06:16.564834 13816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
I20260924 05:06:16.576622 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } attempt: 3
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:06:16.583298 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.577094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.0.98.61:34471)
0924 05:06:16.577163 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.577166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.577179 (+ 13us) server_negotiation.cc:407] Connection header received
0924 05:06:16.577257 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.577258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.577270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.577316 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.577616 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.578240 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.578912 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.579126 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.579181 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.581016 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.581017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.581017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.581024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.582313 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.582667 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.582668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.582668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.582711 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.582877 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.582878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.582878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.582972 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.583014 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.583159 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.583232 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:06:16.583330 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.577053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47872)
0924 05:06:16.577103 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.577116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.577177 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.577327 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.577328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.577481 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.577544 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.577546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.578317 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.578319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.578836 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.578838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.579186 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.579187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.579260 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.579675 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.579681 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.580927 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.582390 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.582391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.582391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.582602 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.582762 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.582763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.582764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.582828 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.583021 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.583022 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.583073 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.583136 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.583235 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 05:06:16.583734 13741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:06:16.584059 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.583949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44202)
0924 05:06:16.583997 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.584032 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:06:16.584357 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.584276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44210)
0924 05:06:16.584313 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.584338 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:06:16.584437 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
W20260924 05:06:16.595252 13823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:16.595480 13823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:16.595501 13823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:16.596647 13823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:16.596688 13823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:16.596701 13823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:16.596724 13823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:16.596747 13823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:16.598294 13823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38405,127.0.98.61:34471,127.0.98.60:45021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:16.598533 13823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:16.598740 13823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:16.603150 13828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:16.603137 13829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:16.603164 13831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:16.603338 13823 server_base.cc:1034] running on GCE node
I20260924 05:06:16.603586 13823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:16.603850 13823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:16.604981 13823 hybrid_clock.cc:648] HybridClock initialized: now 1790226376604967 us; error 30 us; skew 500 ppm
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:16.607232 13823 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:06:16.607640 13823 webserver.cc:460] Webserver started at http://127.0.98.60:43077/ using document root <none> and password file <none>
I20260924 05:06:16.607852 13823 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:16.607900 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:16.608021 13823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:16.609079 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data/instance:
uuid: "6512f497c62f427ba86b0342fd2ac5d6"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "74be6a4cb7c90054a59d28adf6868a6b"
server_key_iv: "af4fe70763619c0358f81daa7bd279d3"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.609405 13823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal/instance:
uuid: "6512f497c62f427ba86b0342fd2ac5d6"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "74be6a4cb7c90054a59d28adf6868a6b"
server_key_iv: "af4fe70763619c0358f81daa7bd279d3"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.610642 13823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:06:16.611263 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:16.611404 13823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:06:16.611482 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal
uuid: "6512f497c62f427ba86b0342fd2ac5d6"
format_stamp: "Formatted at 2026-09-24 05:06:16 on dist-test-slave-m01t"
server_key: "74be6a4cb7c90054a59d28adf6868a6b"
server_key_iv: "af4fe70763619c0358f81daa7bd279d3"
server_key_version: "encryptionkey@0"
I20260924 05:06:16.611541 13823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:16.615574 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } attempt: 1
I20260924 05:06:16.615952 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.615820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44224)
0924 05:06:16.615870 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.615919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:06:16.616238 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.616160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44234)
0924 05:06:16.616194 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.616217 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 05:06:16.616333 13741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
I20260924 05:06:16.632889 13823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:16.633005 13816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } attempt: 2
I20260924 05:06:16.633420 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.633299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44238)
0924 05:06:16.633342 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.633384 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 05:06:16.633760 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.633647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44250)
0924 05:06:16.633692 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.633737 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 05:06:16.633848 13816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45021: connect: Connection refused (error 111)
I20260924 05:06:16.635124 13823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:16.635237 13823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:16.639111 13823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45021
I20260924 05:06:16.639150 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45021 every 8 connection(s)
I20260924 05:06:16.639474 13823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:06:16.640288 13891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:16.641841 13891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:16.648875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13823
I20260924 05:06:16.648981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226228698807-392-0/minicluster-data/master-2/wal/instance
I20260924 05:06:16.649179 392 external_mini_cluster.cc:1428] Setting key 5e9440669de32a7e8fb70287dcaca041
I20260924 05:06:16.650306 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.642539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.0.98.62:38405)
0924 05:06:16.642599 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.642601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.642816 (+ 215us) server_negotiation.cc:407] Connection header received
0924 05:06:16.643298 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.643300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.643320 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.643399 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.644150 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.644623 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.645522 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.645692 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.645769 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.647622 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.647623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.647623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.647629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.648829 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.649257 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.649258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.649258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.649312 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.649574 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.649575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.649575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.649739 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.649803 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.650172 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.650242 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:06:16.650525 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.642512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38405 (local address 127.0.0.1:58452)
0924 05:06:16.642685 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.642739 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.643185 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.643545 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.643550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.643870 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.644071 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.644076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.644687 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.644688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.645465 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.645467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.645768 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.645769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.645872 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.646271 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.646279 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.647542 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.648895 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.648901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.648903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.649180 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.649371 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.649372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.649373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.649513 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.649825 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.649831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.650020 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.650138 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.650298 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":5,"thread_start_us":88,"threads_started":1}
I20260924 05:06:16.651017 13891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:06:16.657330 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.649537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.60:45021)
0924 05:06:16.649702 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.649705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.649742 (+ 37us) server_negotiation.cc:407] Connection header received
0924 05:06:16.649795 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.649797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.649810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.649877 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.650563 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.651491 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.652226 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.652529 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.652578 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.654926 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.654934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.654935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.654942 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.656282 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.656590 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.656594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.656594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.656645 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.656800 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.656801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.656802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.656950 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.657084 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.657150 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.657222 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260924 05:06:16.657394 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.651294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.0.98.61:34471)
0924 05:06:16.651354 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.651355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.651367 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:06:16.651414 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.651415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.651424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.651477 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.651711 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.652149 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.652867 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.653147 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.653202 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.655267 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.655268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.655269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.655276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.656457 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.656747 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.656748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.656748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.656800 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.656952 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.656953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.656953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.657060 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.657110 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.657239 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.657305 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:06:16.657424 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.651238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34471 (local address 127.0.0.1:47888)
0924 05:06:16.651334 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.651342 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.651373 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.651480 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.651481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.651603 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.651651 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.651653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.652213 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.652213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.652771 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.652773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.653214 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.653215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.653280 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.653553 (+ 273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.653558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.655193 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.656530 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.656531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.656531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.656682 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.656850 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.656850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.656851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.656903 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.657109 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.657109 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.657149 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.657209 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.657305 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 05:06:16.657892 13891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:16.664168 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.658114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44262)
0924 05:06:16.658144 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.658154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.658191 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.658309 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.658309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.658414 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.658444 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.658445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.659118 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.659119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.659756 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.659758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.660138 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.660139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.660185 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.660556 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.660560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.661804 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.663263 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.663264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.663265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.663417 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.663570 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.663571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.663571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.663621 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.663825 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.663826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.663862 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.663919 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.664011 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 05:06:16.664175 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.658143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.98.60:45021)
0924 05:06:16.658170 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.658171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.658181 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:06:16.658253 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.658254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.658262 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.658308 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.658502 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.659058 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.659824 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.660083 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.660129 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.661869 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.661870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.661870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.661877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.663193 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.663472 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.663472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.663473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.663520 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.663668 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.663669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.663669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.663777 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.663829 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.663944 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.664052 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":14}
I20260924 05:06:16.664660 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.658077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.0.98.62:38405)
0924 05:06:16.658144 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.658146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.658154 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:16.658177 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.658178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.658188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.658237 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.658722 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.659190 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.659966 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.660152 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.660213 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.662430 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.662432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.662432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.662439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.663605 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.663884 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.663885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.663886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.663935 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.664085 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.664086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.664086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.664190 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.664378 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.664514 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.664599 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:06:16.665966 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: Bootstrap starting.
I20260924 05:06:16.666929 13891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:16.667203 13891 log.cc:824] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:16.667979 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: No bootstrap required, opened a new log
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:06:16.669435 13891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.669570 13891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:16.669615 13891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6512f497c62f427ba86b0342fd2ac5d6, State: Initialized, Role: FOLLOWER
I20260924 05:06:16.669809 13891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.670133 13891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:16.670421 13897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.671279 13897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:06:16.671922 13908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:06:16.672013 13908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:06:16.672266 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.665363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.0.98.61:34471)
0924 05:06:16.665405 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.665406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.665414 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:16.665436 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.665437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.665446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.665506 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.665951 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.666633 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.667275 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.667532 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.667587 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.669551 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.669552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.669553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.669560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.671109 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.671486 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.671487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.671488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.671544 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.671749 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.671750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.671751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.671888 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.672065 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.672106 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.672180 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:06:16.673815 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:06:16.672631 13823 master_runner.cc:186] Error getting master registration for 127.0.98.62:38405: OK, instance_id { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" instance_seqno: 1790226376350594 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:06:16.674670 13823 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:06:16.681870 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.675325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.0.98.61:34471)
0924 05:06:16.675359 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.675361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.675549 (+ 188us) server_negotiation.cc:407] Connection header received
0924 05:06:16.675603 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.675603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.675612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.675662 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.676156 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.676676 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.677366 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.677596 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.677648 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.678976 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.678977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.678977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.678983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.680382 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.680729 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.680730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.680731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.680783 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.680941 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.680942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.680942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.681045 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.681142 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.681395 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.681468 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:06:16.684370 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.675212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.98.60:45021)
0924 05:06:16.675294 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.675296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.675570 (+ 274us) server_negotiation.cc:407] Connection header received
0924 05:06:16.675660 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.675661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.675670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.675697 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.676396 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.677095 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.678437 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.678718 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.678757 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.680514 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.680515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.680516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.680523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.682352 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.683583 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.683584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.683584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.683646 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.683856 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.683857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.683858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.684008 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.684109 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.684210 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.684280 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 05:06:16.684495 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.675254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.0.98.62:38405)
0924 05:06:16.675324 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.675326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.677867 (+ 2541us) server_negotiation.cc:407] Connection header received
0924 05:06:16.677949 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.677951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.677961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.678008 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.678597 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.679266 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.680163 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.680460 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.680496 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.681994 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.681995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.681995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.682002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.683303 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.683649 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.683650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.683650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.683702 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.683899 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.683908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.683909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.684041 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.684139 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.684318 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.684397 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 05:06:16.691493 13741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } attempt: 2
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:06:16.698434 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.691791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44288)
0924 05:06:16.691853 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.691867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.691922 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.692048 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.692049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.692256 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.692308 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.692310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.692910 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.692911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.693734 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.693737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.694249 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.694250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.694304 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.694768 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.694773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.695894 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.697556 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.697557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.697557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.697748 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.697928 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.697929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.697929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.697980 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.698207 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.698208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.698245 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.698288 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.698364 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 05:06:16.698488 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.691850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.0.98.60:45021)
0924 05:06:16.691890 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.691892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.691901 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:06:16.691981 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.691982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.691990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.692034 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.692371 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.692838 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.693825 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.694142 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.694206 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.695960 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.695961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.695962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.695969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.697487 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.697815 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.697816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.697816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.697875 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.698027 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.698027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.698028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.698158 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.698221 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.698316 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.698393 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:06:16.700058 13741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a: Bootstrap starting.
I20260924 05:06:16.700794 13741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:16.701140 13741 log.cc:824] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:16.702004 13741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a: No bootstrap required, opened a new log
I20260924 05:06:16.703962 13741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.704113 13741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:16.704154 13741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493c3d3d070b46ce905a0d3eb05ceb4a, State: Initialized, Role: FOLLOWER
I20260924 05:06:16.704275 13741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.704535 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.704591 13741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:16.704614 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:06:16.706346 13929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:06:16.706458 13929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:06:16.706977 13673 master_runner.cc:186] Error getting master registration for 127.0.98.61:34471: OK, instance_id { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" instance_seqno: 1790226376490473 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:06:16.709192 13673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:06:16.724997 13816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } attempt: 3
Sep 24 05:06:16 dist-test-slave-m01t krb5kdc[13660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:06:16.733523 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.725353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45021 (local address 127.0.0.1:44302)
0924 05:06:16.725414 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:16.725425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:16.725464 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:16.725621 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:16.725622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:16.725852 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:16.725909 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.725912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.726840 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.726841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.727720 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:16.727724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.728206 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.728207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:16.728270 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.729005 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:16.729010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:16.730623 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:16.732412 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.732414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.732414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.732637 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.732854 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:16.732855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:16.732855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:16.732928 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:16.733190 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:16.733191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:16.733242 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 05:06:16.733308 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 05:06:16.733414 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:06:16.733522 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:16.725414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.0.98.60:45021)
0924 05:06:16.725458 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:16.725460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:16.725471 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:06:16.725526 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:16.725527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:16.725537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:16.725606 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:16.725996 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.726746 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.727816 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:16.728134 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:16.728199 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:16.730727 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:16.730729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:16.730730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:16.730738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:16.732325 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.732710 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.732711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.732712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.732776 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:16.732984 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:16.732985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:16.732985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:16.733129 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:16.733197 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:16.733341 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:06:16.733421 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:06:16.735517 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: Bootstrap starting.
I20260924 05:06:16.735724 13918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:16.735849 13918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.736140 13918 leader_election.cc:288] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac942bd5dd44acd8a5da70949118062 (127.0.98.61:34471), 6512f497c62f427ba86b0342fd2ac5d6 (127.0.98.60:45021)
I20260924 05:06:16.736294 13816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:16.736474 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6512f497c62f427ba86b0342fd2ac5d6" is_pre_election: true
I20260924 05:06:16.736614 13865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493c3d3d070b46ce905a0d3eb05ceb4a in term 0.
I20260924 05:06:16.736604 13816 log.cc:824] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:16.736833 13692 leader_election.cc:302] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493c3d3d070b46ce905a0d3eb05ceb4a, 6512f497c62f427ba86b0342fd2ac5d6; no voters:
I20260924 05:06:16.737174 13918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:06:16.737228 13918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:06:16.737249 13918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:16.737851 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: No bootstrap required, opened a new log
I20260924 05:06:16.738268 13918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.738394 13918 leader_election.cc:288] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [CANDIDATE]: Term 1 election: Requested vote from peers eac942bd5dd44acd8a5da70949118062 (127.0.98.61:34471), 6512f497c62f427ba86b0342fd2ac5d6 (127.0.98.60:45021)
I20260924 05:06:16.738744 13865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6512f497c62f427ba86b0342fd2ac5d6"
I20260924 05:06:16.738837 13865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:16.739629 13865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493c3d3d070b46ce905a0d3eb05ceb4a in term 1.
I20260924 05:06:16.739837 13692 leader_election.cc:302] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493c3d3d070b46ce905a0d3eb05ceb4a, 6512f497c62f427ba86b0342fd2ac5d6; no voters:
I20260924 05:06:16.739852 13816 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.740007 13816 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:16.740043 13816 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac942bd5dd44acd8a5da70949118062, State: Initialized, Role: FOLLOWER
I20260924 05:06:16.740155 13816 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.740172 13918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:16.740363 13918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [term 1 LEADER]: Becoming Leader. State: Replica: 493c3d3d070b46ce905a0d3eb05ceb4a, State: Running, Role: LEADER
I20260924 05:06:16.740446 13918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } }
I20260924 05:06:16.740499 13816 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:16.740803 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493c3d3d070b46ce905a0d3eb05ceb4a. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.740877 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:16.741053 13789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac942bd5dd44acd8a5da70949118062"
I20260924 05:06:16.741150 13789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:16.741688 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac942bd5dd44acd8a5da70949118062" is_pre_election: true
I20260924 05:06:16.741987 13789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493c3d3d070b46ce905a0d3eb05ceb4a in term 1.
I20260924 05:06:16.742329 13936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:16.742558 13936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:16.744131 13865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [term 1 FOLLOWER]: Refusing update from remote peer 493c3d3d070b46ce905a0d3eb05ceb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:16.744494 13932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:16.744645 13789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [term 1 FOLLOWER]: Refusing update from remote peer 493c3d3d070b46ce905a0d3eb05ceb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:16.744864 13918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:16.746100 13897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493c3d3d070b46ce905a0d3eb05ceb4a. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.746170 13897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:06:16.747391 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.747474 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:16.747545 13947 mvcc.cc:205] Tried to move back new op lower bound from 7332767239142010880 to 7332767239129346048. Current Snapshot: MvccSnapshot[applied={T|T < 7332767239142010880}]
I20260924 05:06:16.747656 13897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.747745 13897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:06:16.747893 13949 mvcc.cc:205] Tried to move back new op lower bound from 7332767239142010880 to 7332767239129346048. Current Snapshot: MvccSnapshot[applied={T|T < 7332767239142010880}]
I20260924 05:06:16.748095 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.748186 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:06:16.748235 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493c3d3d070b46ce905a0d3eb05ceb4a. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.748281 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:06:16.750296 13946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:06:16.750414 13946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:06:16.750878 13936 catalog_manager.cc:1347] Generated new cluster ID: dcb769b0fbca4a5d8a3790357bd4d7f3
I20260924 05:06:16.750936 13936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:16.757896 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.758018 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:16.758607 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493c3d3d070b46ce905a0d3eb05ceb4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38405 } } peers { permanent_uuid: "eac942bd5dd44acd8a5da70949118062" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34471 } } peers { permanent_uuid: "6512f497c62f427ba86b0342fd2ac5d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45021 } } }
I20260924 05:06:16.758709 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:06:16.762427 13950 mvcc.cc:205] Tried to move back new op lower bound from 7332767239142010880 to 7332767239129346048. Current Snapshot: MvccSnapshot[applied={T|T < 7332767239142010880}]
I20260924 05:06:16.764544 13936 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:06:16.765262 13936 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:16.772508 13936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493c3d3d070b46ce905a0d3eb05ceb4a: Generated new TSK 0
I20260924 05:06:16.773905 13936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:06:17.672447 13908 catalog_manager.cc:1259] Loaded cluster ID: dcb769b0fbca4a5d8a3790357bd4d7f3
I20260924 05:06:17.672518 13908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: loading cluster ID for follower catalog manager: success
I20260924 05:06:17.673297 13908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: acquiring CA information for follower catalog manager: success
I20260924 05:06:17.673568 13908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6512f497c62f427ba86b0342fd2ac5d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:06:17.750852 13946 catalog_manager.cc:1259] Loaded cluster ID: dcb769b0fbca4a5d8a3790357bd4d7f3
I20260924 05:06:17.750928 13946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: loading cluster ID for follower catalog manager: success
I20260924 05:06:17.751717 13946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: acquiring CA information for follower catalog manager: success
I20260924 05:06:17.752005 13946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eac942bd5dd44acd8a5da70949118062: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:06:17.833587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13673
I20260924 05:06:17.838240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13748
I20260924 05:06:17.842670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13823
2026-09-24T05:06:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2663 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:06:17 dist-test-slave-m01t krb5kdc[13965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:06:17 dist-test-slave-m01t krb5kdc[13965](info): set up 2 sockets
Sep 24 05:06:17 dist-test-slave-m01t krb5kdc[13965](info): commencing operation
krb5kdc: starting...
W20260924 05:06:18.785387 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:06:18 dist-test-slave-m01t krb5kdc[13965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:06:18.892043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:18.970332 13978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:18.970544 13978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:18.970566 13978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:18.971698 13978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:18.971743 13978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:18.971756 13978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:18.971768 13978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:18.971791 13978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:18.973233 13978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:18.973469 13978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:18.973678 13978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:06:18.978124 13978 server_base.cc:1034] running on GCE node
W20260924 05:06:18.978111 13983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:18.978106 13986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:18.978101 13984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:18.978452 13978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:18.978726 13978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:18.979867 13978 hybrid_clock.cc:648] HybridClock initialized: now 1790226378979851 us; error 28 us; skew 500 ppm
Sep 24 05:06:18 dist-test-slave-m01t krb5kdc[13965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:18.982120 13978 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:18.982547 13978 webserver.cc:460] Webserver started at http://127.0.98.62:40755/ using document root <none> and password file <none>
I20260924 05:06:18.982739 13978 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:18.982784 13978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:18.982901 13978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:18.983902 13978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "992d234be1e643c381f675a90cde62f5"
format_stamp: "Formatted at 2026-09-24 05:06:18 on dist-test-slave-m01t"
server_key: "23b82cb2a498a29bfd1ca241b907870c"
server_key_iv: "d1e35d7ee6f13cd99128b9166b593b5e"
server_key_version: "encryptionkey@0"
I20260924 05:06:18.984220 13978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "992d234be1e643c381f675a90cde62f5"
format_stamp: "Formatted at 2026-09-24 05:06:18 on dist-test-slave-m01t"
server_key: "23b82cb2a498a29bfd1ca241b907870c"
server_key_iv: "d1e35d7ee6f13cd99128b9166b593b5e"
server_key_version: "encryptionkey@0"
I20260924 05:06:18.985387 13978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:06:18.986039 13993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:18.986181 13978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:18.986258 13978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "992d234be1e643c381f675a90cde62f5"
format_stamp: "Formatted at 2026-09-24 05:06:18 on dist-test-slave-m01t"
server_key: "23b82cb2a498a29bfd1ca241b907870c"
server_key_iv: "d1e35d7ee6f13cd99128b9166b593b5e"
server_key_version: "encryptionkey@0"
I20260924 05:06:18.986320 13978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:18.999612 13978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:19.001286 13978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:19.001430 13978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:19.005168 13978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37897
I20260924 05:06:19.005203 14045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37897 every 8 connection(s)
I20260924 05:06:19.005534 13978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:19.005988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 13978
I20260924 05:06:19.006072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:06:19.006245 392 external_mini_cluster.cc:1428] Setting key 099206988eb288b1d736886b932dad26
I20260924 05:06:19.006378 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:19.009287 14046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5: Bootstrap starting.
I20260924 05:06:19.009976 14046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:19.010252 14046 log.cc:824] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:19.010849 14046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5: No bootstrap required, opened a new log
Sep 24 05:06:19 dist-test-slave-m01t krb5kdc[13965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:19.012894 14046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992d234be1e643c381f675a90cde62f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37897 } }
I20260924 05:06:19.013080 14046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:19.013126 14046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992d234be1e643c381f675a90cde62f5, State: Initialized, Role: FOLLOWER
I20260924 05:06:19.013244 14046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992d234be1e643c381f675a90cde62f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37897 } }
I20260924 05:06:19.013320 14046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:19.013389 14046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:19.013459 14046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:19.014114 14046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992d234be1e643c381f675a90cde62f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37897 } }
I20260924 05:06:19.014212 14046 leader_election.cc:302] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 992d234be1e643c381f675a90cde62f5; no voters:
I20260924 05:06:19.014398 14046 leader_election.cc:288] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:06:19.014438 14051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:19.014575 14051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [term 1 LEADER]: Becoming Leader. State: Replica: 992d234be1e643c381f675a90cde62f5, State: Running, Role: LEADER
I20260924 05:06:19.014657 14046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:19.014676 14051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992d234be1e643c381f675a90cde62f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37897 } }
I20260924 05:06:19.015009 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992d234be1e643c381f675a90cde62f5. Latest consensus state: current_term: 1 leader_uuid: "992d234be1e643c381f675a90cde62f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992d234be1e643c381f675a90cde62f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37897 } } }
I20260924 05:06:19.015224 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:19.015159 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "992d234be1e643c381f675a90cde62f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992d234be1e643c381f675a90cde62f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37897 } } }
I20260924 05:06:19.015585 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5 [sys.catalog]: This master's current role is: LEADER
W20260924 05:06:19.016316 14067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:06:19.016417 14067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:06:19.016474 14060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:19.016645 14060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:19.016644 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:19.006717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.0.98.62:37897)
0924 05:06:19.006881 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:19.006891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:19.006915 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:06:19.007390 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:19.007397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:19.007444 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:19.007551 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:19.008246 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:19.009105 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:19.010119 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:19.010542 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:19.010649 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:19.013074 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:19.013077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:19.013078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:19.013088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:19.014880 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:19.015262 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:19.015265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:19.015265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:19.015361 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:19.015563 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:19.015564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:19.015565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:19.015864 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:19.016001 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:19.016274 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 05:06:19.016418 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260924 05:06:19.017511 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:06:19.018060 14060 catalog_manager.cc:1347] Generated new cluster ID: 140873f0b38248629872f2f4fda0e0f5
I20260924 05:06:19.018106 14060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:19.042464 14060 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:06:19.043051 14060 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:19.046072 14060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 992d234be1e643c381f675a90cde62f5: Generated new TSK 0
I20260924 05:06:19.046241 14060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:06:19.071197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 13978
2026-09-24T05:06:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1227 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:06:19 dist-test-slave-m01t krb5kdc[14073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:06:19 dist-test-slave-m01t krb5kdc[14073](info): set up 2 sockets
Sep 24 05:06:19 dist-test-slave-m01t krb5kdc[14073](info): commencing operation
krb5kdc: starting...
W20260924 05:06:20.013441 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:06:20.118551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:20.196852 14086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:20.197062 14086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:20.197083 14086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:20.198237 14086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:20.198275 14086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:20.198287 14086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:20.198310 14086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:20.198329 14086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:20.199736 14086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:20.199959 14086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:20.200152 14086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:06:20.204556 14086 server_base.cc:1034] running on GCE node
W20260924 05:06:20.204535 14091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.204528 14092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.204517 14094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:20.204957 14086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:20.205205 14086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:20.206341 14086 hybrid_clock.cc:648] HybridClock initialized: now 1790226380206328 us; error 27 us; skew 500 ppm
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:20.208593 14086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:20.208961 14086 webserver.cc:460] Webserver started at http://127.0.98.62:36431/ using document root <none> and password file <none>
I20260924 05:06:20.209132 14086 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:20.209169 14086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:20.209254 14086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:20.210240 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "324ab9e684084869b2a6130548cb31e1"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "822b391c141b20489712eb10b56b301b"
server_key_iv: "03c2e8f6030585a00760e8dc0ec24310"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.210518 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "324ab9e684084869b2a6130548cb31e1"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "822b391c141b20489712eb10b56b301b"
server_key_iv: "03c2e8f6030585a00760e8dc0ec24310"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.211609 14086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:06:20.212203 14101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.212347 14086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:20.212430 14086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "324ab9e684084869b2a6130548cb31e1"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "822b391c141b20489712eb10b56b301b"
server_key_iv: "03c2e8f6030585a00760e8dc0ec24310"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.212500 14086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:20.223227 14086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:20.224891 14086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:20.224998 14086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:20.228564 14086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33755
I20260924 05:06:20.228607 14153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33755 every 8 connection(s)
I20260924 05:06:20.228921 14086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:20.229911 14154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:20.232163 14154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap starting.
I20260924 05:06:20.232721 14154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:20.232822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14086
I20260924 05:06:20.232946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:06:20.233006 14154 log.cc:824] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:20.233129 392 external_mini_cluster.cc:1428] Setting key a80113363e310a62bd38c13a9f411a31
I20260924 05:06:20.233767 14154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: No bootstrap required, opened a new log
I20260924 05:06:20.236132 14154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.236320 14154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:20.236363 14154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324ab9e684084869b2a6130548cb31e1, State: Initialized, Role: FOLLOWER
I20260924 05:06:20.236493 14154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.236568 14154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:20.236603 14154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:20.236644 14154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:20.237254 14154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.237354 14154 leader_election.cc:302] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324ab9e684084869b2a6130548cb31e1; no voters:
I20260924 05:06:20.237509 14154 leader_election.cc:288] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:06:20.237541 14159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:20.237700 14159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 LEADER]: Becoming Leader. State: Replica: 324ab9e684084869b2a6130548cb31e1, State: Running, Role: LEADER
I20260924 05:06:20.237833 14154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:20.237862 14159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.238180 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324ab9e684084869b2a6130548cb31e1. Latest consensus state: current_term: 1 leader_uuid: "324ab9e684084869b2a6130548cb31e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } } }
I20260924 05:06:20.238248 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: This master's current role is: LEADER
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:20.239279 14160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "324ab9e684084869b2a6130548cb31e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } } }
I20260924 05:06:20.240003 14160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: This master's current role is: LEADER
W20260924 05:06:20.239584 14175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:06:20.240063 14175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:06:20.240146 14163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:20.240339 14163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:20.242369 14163 catalog_manager.cc:1347] Generated new cluster ID: 9c6a13e021d1444fba2b8edcce9f677a
I20260924 05:06:20.242416 14163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:20.243602 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.233595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.0.98.62:33755)
0924 05:06:20.233774 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.233782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.233807 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:06:20.234241 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.234248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.234298 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.234420 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:20.235142 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.235978 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.236987 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.237432 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.237527 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.240216 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:20.240219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:20.240220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:20.240232 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:20.241956 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.242324 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.242327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.242328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.242437 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.242644 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.242645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.242645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.242914 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:20.243025 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.243255 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.243385 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260924 05:06:20.260617 14163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:06:20.261143 14163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:20.266552 14163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Generated new TSK 0
I20260924 05:06:20.266721 14163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:06:20.294651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33755
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:20.372668 14180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:20.372861 14180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:20.372884 14180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:20.374024 14180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:20.374063 14180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:20.374090 14180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:20.375506 14180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:20.375725 14180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:20.375949 14180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:20.380498 14186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.380517 14185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.380517 14188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:20.380692 14180 server_base.cc:1034] running on GCE node
I20260924 05:06:20.380869 14180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:20.381079 14180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:20.382213 14180 hybrid_clock.cc:648] HybridClock initialized: now 1790226380382198 us; error 26 us; skew 500 ppm
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:20.384529 14180 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:20.384927 14180 webserver.cc:460] Webserver started at http://127.0.98.1:35739/ using document root <none> and password file <none>
I20260924 05:06:20.385123 14180 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:20.385162 14180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:20.385273 14180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:20.386279 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "9e968d05c3e745989494cda4f5ef35a2"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "c24860797dc5dcd8d24ea9485404689e"
server_key_iv: "bfedb11424d6bd0c63172769dc30cf19"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.386588 14180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9e968d05c3e745989494cda4f5ef35a2"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "c24860797dc5dcd8d24ea9485404689e"
server_key_iv: "bfedb11424d6bd0c63172769dc30cf19"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.387735 14180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:06:20.388370 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.388523 14180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:20.388605 14180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "9e968d05c3e745989494cda4f5ef35a2"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "c24860797dc5dcd8d24ea9485404689e"
server_key_iv: "bfedb11424d6bd0c63172769dc30cf19"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.388674 14180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:20.406997 14180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:20.409029 14180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:20.409157 14180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:20.409373 14180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:20.409651 14180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:20.409683 14180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.409703 14180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:20.409793 14180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.415102 14180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32871
I20260924 05:06:20.415154 14308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32871 every 8 connection(s)
I20260924 05:06:20.415444 14180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:20.418821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14180
I20260924 05:06:20.418918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:06:20.419094 392 external_mini_cluster.cc:1428] Setting key e8624a5357eff6f2f86483627e2e42b4
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:20.424417 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.416040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33355 (local address 127.0.98.62:33755)
0924 05:06:20.416105 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.416108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.416272 (+ 164us) server_negotiation.cc:407] Connection header received
0924 05:06:20.416740 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.416741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.416751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.416789 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:20.417633 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.418147 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.419131 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.419429 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.419488 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.421699 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:20.421701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:20.421701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:20.421736 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:20.423040 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.423513 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.423513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.423514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.423563 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.423811 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.423811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.423812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.423918 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:20.423963 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.424280 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.424344 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:06:20.424702 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.416019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:33355)
0924 05:06:20.416165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.416199 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.416655 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.416849 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.416852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.417381 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:20.417567 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.417572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.418209 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.418209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.419054 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.419057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.419511 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.419512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.419657 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.420169 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:20.420177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:20.421607 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:20.423129 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:20.423137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:20.423140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:20.423443 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:20.423616 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:20.423616 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:20.423617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:20.423753 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:20.423970 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:20.423977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:20.424153 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.424274 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.424460 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260924 05:06:20.425240 14309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33755
I20260924 05:06:20.425328 14309 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:20.425503 14309 heartbeater.cc:511] Master 127.0.98.62:33755 requested a full tablet report, sending...
I20260924 05:06:20.426045 14118 ts_manager.cc:194] Registered new tserver with Master: 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871)
I20260924 05:06:20.426652 14118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33355
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:06:20.441619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33755
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:20.518250 14314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:20.518455 14314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:20.518476 14314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:20.519603 14314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:20.519646 14314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:20.519675 14314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:20.521126 14314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:20.521354 14314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:20.521565 14314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:20.526216 14322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.526208 14320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.526208 14319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:20.526449 14314 server_base.cc:1034] running on GCE node
I20260924 05:06:20.526692 14314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:20.526928 14314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:20.528074 14314 hybrid_clock.cc:648] HybridClock initialized: now 1790226380528054 us; error 30 us; skew 500 ppm
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:20.530485 14314 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:20.530901 14314 webserver.cc:460] Webserver started at http://127.0.98.2:45739/ using document root <none> and password file <none>
I20260924 05:06:20.531093 14314 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:20.531133 14314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:20.531225 14314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:20.532233 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "4419fa25de374c7ebb218618cdc7f720"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "d6655da95818b2fac8809b48ea1088df"
server_key_iv: "188c34843d2e5a7831ac9b3bddaa763e"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.532548 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4419fa25de374c7ebb218618cdc7f720"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "d6655da95818b2fac8809b48ea1088df"
server_key_iv: "188c34843d2e5a7831ac9b3bddaa763e"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.533785 14314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:06:20.534458 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.534597 14314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:20.534675 14314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "4419fa25de374c7ebb218618cdc7f720"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "d6655da95818b2fac8809b48ea1088df"
server_key_iv: "188c34843d2e5a7831ac9b3bddaa763e"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.534730 14314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:20.548444 14314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:20.550524 14314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:20.550634 14314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:20.550812 14314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:20.551091 14314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:20.551126 14314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.551147 14314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:20.551169 14314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.556393 14314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39561
I20260924 05:06:20.556434 14442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39561 every 8 connection(s)
I20260924 05:06:20.556723 14314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:20.564625 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.557361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48557 (local address 127.0.98.62:33755)
0924 05:06:20.557420 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.557422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.557604 (+ 182us) server_negotiation.cc:407] Connection header received
0924 05:06:20.558121 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.558122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.558130 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.558184 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:20.559068 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.559553 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.560489 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.560664 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.560717 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.562622 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:20.562624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:20.562624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:20.562631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:20.563643 (+ 1012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.563961 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.563962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.563962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.564003 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.564167 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.564168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.564168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.564259 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:20.564301 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.564501 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.564558 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:06:20.564795 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.557329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:48557)
0924 05:06:20.557485 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.557526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.558039 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.558241 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.558245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.558802 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:20.558999 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.559004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.559617 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.559618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.560428 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.560430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.560718 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.560719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.560811 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.561217 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:20.561225 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:20.562540 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:20.563709 (+ 1169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:20.563714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:20.563716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:20.563913 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:20.564044 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:20.564045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:20.564045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:20.564128 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:20.564298 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:20.564300 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:20.564411 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.564483 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.564593 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 05:06:20.565238 14443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33755
I20260924 05:06:20.565336 14443 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:20.565505 14443 heartbeater.cc:511] Master 127.0.98.62:33755 requested a full tablet report, sending...
I20260924 05:06:20.565730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14314
I20260924 05:06:20.565851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:06:20.565878 14118 ts_manager.cc:194] Registered new tserver with Master: 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561)
I20260924 05:06:20.566035 392 external_mini_cluster.cc:1428] Setting key fc4f7783723298d0e2aab162c03aa2f5
I20260924 05:06:20.566318 14118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48557
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:06:20.586170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33755
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:20.665189 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:20.665408 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:20.665431 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:20.666666 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:20.666718 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:20.666749 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:20.668205 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:20.668435 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:20.668651 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:20.673187 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.673231 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.673185 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:20.673445 14448 server_base.cc:1034] running on GCE node
I20260924 05:06:20.673616 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:20.673866 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:20.675016 14448 hybrid_clock.cc:648] HybridClock initialized: now 1790226380675001 us; error 26 us; skew 500 ppm
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:20.677332 14448 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:20.677768 14448 webserver.cc:460] Webserver started at http://127.0.98.3:45527/ using document root <none> and password file <none>
I20260924 05:06:20.677991 14448 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:20.678040 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:20.678172 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:20.679177 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d1c348e9d3340b2b56303fea2e370be"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "6c397102699dfb89fea3e45f25dbc6b5"
server_key_iv: "221e2f3ac30b5df10447971c7d5c07ed"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.679498 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d1c348e9d3340b2b56303fea2e370be"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "6c397102699dfb89fea3e45f25dbc6b5"
server_key_iv: "221e2f3ac30b5df10447971c7d5c07ed"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.680714 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:06:20.681335 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.681485 14448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.681555 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "6d1c348e9d3340b2b56303fea2e370be"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "6c397102699dfb89fea3e45f25dbc6b5"
server_key_iv: "221e2f3ac30b5df10447971c7d5c07ed"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.681618 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:20.691646 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:20.693655 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:20.693812 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:20.694025 14448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:20.694304 14448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:20.694336 14448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.694365 14448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:20.694386 14448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.700680 14448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44833
I20260924 05:06:20.700714 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44833 every 8 connection(s)
I20260924 05:06:20.701125 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:20.708835 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.701813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44225 (local address 127.0.98.62:33755)
0924 05:06:20.701862 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.701864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.702073 (+ 209us) server_negotiation.cc:407] Connection header received
0924 05:06:20.702579 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.702580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.702593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.702662 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:20.703501 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.703974 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.704854 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.705026 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.705066 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.706878 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:20.706879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:20.706879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:20.706886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:20.707883 (+ 997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.708192 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.708192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.708193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.708233 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.708406 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.708406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.708407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.708493 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:20.708534 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.708723 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.708772 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:06:20.709036 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.701830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:44225)
0924 05:06:20.701965 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.701998 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.702481 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.702714 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.702718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.703249 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:20.703435 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.703440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.704030 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.704030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.704793 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.704795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.705074 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.705074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.705162 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.705515 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:20.705523 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:20.706787 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:20.707940 (+ 1153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:20.707944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:20.707946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:20.708140 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:20.708275 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:20.708275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:20.708276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:20.708361 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:20.708533 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:20.708535 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:20.708637 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.708708 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.708821 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 05:06:20.709556 14577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33755
I20260924 05:06:20.709630 14577 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:20.709790 14577 heartbeater.cc:511] Master 127.0.98.62:33755 requested a full tablet report, sending...
I20260924 05:06:20.710088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14448
I20260924 05:06:20.710136 14117 ts_manager.cc:194] Registered new tserver with Master: 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833)
I20260924 05:06:20.710189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:06:20.710376 392 external_mini_cluster.cc:1428] Setting key 46135b2843b7d1a3d489ce750ff1ec9f
I20260924 05:06:20.710577 14117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44225
I20260924 05:06:20.711215 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:06:20.717602 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.712385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.98.62:33755)
0924 05:06:20.712437 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.712438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.712488 (+ 50us) server_negotiation.cc:407] Connection header received
0924 05:06:20.712547 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.712548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.712557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.712601 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:20.713108 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.713562 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.714346 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.714569 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.714620 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.715654 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:20.715655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:20.715656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:20.715662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:20.716636 (+ 974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.716925 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.716926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.716927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.716977 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:20.717131 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:20.717132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:20.717133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:20.717263 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:20.717354 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.717445 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.717515 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:06:20.718858 14117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:06:20.720302 14117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:06:20.726217 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.722603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.0.98.1:32871)
0924 05:06:20.722805 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.722810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.722819 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:06:20.722857 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.722860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.722909 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.723154 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:20.723225 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.723882 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.725185 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.725850 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.725916 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.725983 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.726088 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.726127 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260924 05:06:20.726205 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.722225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32871 (local address 127.0.0.1:56364)
0924 05:06:20.722563 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.722573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.722611 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.723013 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.723016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.723018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:20.723084 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.723085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.723977 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.723978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.725102 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.725105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.725913 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.725914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.725990 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.726004 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.726073 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.726126 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":59,"threads_started":1}
I20260924 05:06:20.726493 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.722223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44833 (local address 127.0.0.1:60834)
0924 05:06:20.722341 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.722380 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.722437 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.723139 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.723140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.723141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:20.723170 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.723171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.724445 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.724446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.725335 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.725337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.726265 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.726266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.726330 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.726333 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.726361 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.726411 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"spinlock_wait_cycles":896,"thread_start_us":63,"threads_started":1}
I20260924 05:06:20.726544 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.722439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39561 (local address 127.0.0.1:34436)
0924 05:06:20.722771 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.722781 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.722821 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.723043 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.723044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.723046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:20.723098 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.723099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.724399 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.724400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.725372 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.725374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.726316 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.726316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.726367 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.726369 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.726418 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.726450 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":74,"threads_started":1}
I20260924 05:06:20.726583 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.722544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.98.2:39561)
0924 05:06:20.722793 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.722821 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.722832 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:06:20.722873 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.722875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.722889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.722956 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:20.723171 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.724290 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.725456 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.726169 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.726235 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.726289 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.726455 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.726502 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20260924 05:06:20.727025 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.722329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.3:44833)
0924 05:06:20.722737 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.722742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.722753 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:06:20.722790 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.722792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.722822 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.723239 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:20.723306 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.724336 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.725420 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.726169 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.726849 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.726913 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.726936 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.726967 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":104,"threads_started":1}
I20260924 05:06:20.727288 14377 tablet_service.cc:1467] Processing CreateTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e (DEFAULT_TABLE table=test-table [id=7a618bc0816842578f021138c4dcadc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:20.727664 14377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc44b4fd56044edbd36ce01f8dd6d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:20.727643 14511 tablet_service.cc:1467] Processing CreateTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e (DEFAULT_TABLE table=test-table [id=7a618bc0816842578f021138c4dcadc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:20.727932 14511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc44b4fd56044edbd36ce01f8dd6d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:20.730406 14243 tablet_service.cc:1467] Processing CreateTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e (DEFAULT_TABLE table=test-table [id=7a618bc0816842578f021138c4dcadc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:20.730706 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc44b4fd56044edbd36ce01f8dd6d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:20.733062 14593 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap starting.
I20260924 05:06:20.733227 14591 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Bootstrap starting.
I20260924 05:06:20.733860 14593 tablet_bootstrap.cc:654] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:20.733870 14591 tablet_bootstrap.cc:654] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:20.734313 14591 log.cc:824] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:20.734357 14593 log.cc:824] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:20.734665 14592 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Bootstrap starting.
I20260924 05:06:20.735010 14593 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: No bootstrap required, opened a new log
I20260924 05:06:20.735066 14593 ts_tablet_manager.cc:1397] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:06:20.735261 14592 tablet_bootstrap.cc:654] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:20.735702 14592 log.cc:824] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:20.736141 14591 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: No bootstrap required, opened a new log
I20260924 05:06:20.736212 14591 ts_tablet_manager.cc:1397] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:06:20.736431 14592 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: No bootstrap required, opened a new log
I20260924 05:06:20.736495 14592 ts_tablet_manager.cc:1397] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:20.736702 14593 raft_consensus.cc:348] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.736850 14593 raft_consensus.cc:374] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:20.736882 14593 raft_consensus.cc:729] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e968d05c3e745989494cda4f5ef35a2, State: Initialized, Role: FOLLOWER
I20260924 05:06:20.737056 14593 consensus_queue.cc:260] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.737365 14309 heartbeater.cc:503] Master 127.0.98.62:33755 was elected leader, sending a full tablet report...
I20260924 05:06:20.737378 14593 ts_tablet_manager.cc:1428] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:06:20.738435 14592 raft_consensus.cc:348] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.738478 14591 raft_consensus.cc:348] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.738631 14591 raft_consensus.cc:374] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:20.738631 14592 raft_consensus.cc:374] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:20.738670 14592 raft_consensus.cc:729] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1c348e9d3340b2b56303fea2e370be, State: Initialized, Role: FOLLOWER
I20260924 05:06:20.738695 14591 raft_consensus.cc:729] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4419fa25de374c7ebb218618cdc7f720, State: Initialized, Role: FOLLOWER
I20260924 05:06:20.738842 14592 consensus_queue.cc:260] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.738842 14591 consensus_queue.cc:260] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.739121 14592 ts_tablet_manager.cc:1428] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:06:20.739121 14591 ts_tablet_manager.cc:1428] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:06:20.739231 14577 heartbeater.cc:503] Master 127.0.98.62:33755 was elected leader, sending a full tablet report...
I20260924 05:06:20.739236 14443 heartbeater.cc:503] Master 127.0.98.62:33755 was elected leader, sending a full tablet report...
I20260924 05:06:20.756145 14599 raft_consensus.cc:482] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:20.756264 14599 raft_consensus.cc:504] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.756630 14599 leader_election.cc:288] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871)
I20260924 05:06:20.759493 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.756850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45045 (local address 127.0.98.2:39561)
0924 05:06:20.756926 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.756929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.756938 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:06:20.756966 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.756967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.756979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.757033 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:20.757166 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.757905 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.758801 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.759182 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.759241 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.759252 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.759394 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.759431 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:06:20.759516 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.756769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39561 (local address 127.0.98.3:45045)
0924 05:06:20.756830 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.756841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.756882 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.757040 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.757041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.757042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:20.757105 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.757107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.757968 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.757969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.758707 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.758709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.759251 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.759252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.759330 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.759334 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.759377 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.759417 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:06:20.759514 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.756992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48409 (local address 127.0.98.1:32871)
0924 05:06:20.757045 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.757047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.757054 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:06:20.757078 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.757079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.757090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.757134 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:20.757282 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.757999 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.758859 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.759224 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.759278 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.759288 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.759415 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.759449 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:06:20.759531 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.756799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32871 (local address 127.0.98.3:48409)
0924 05:06:20.756926 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.756935 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:20.756975 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:20.757171 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:20.757172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:20.757174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:20.757230 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.757231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.758054 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.758055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.758785 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:20.758787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.759290 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.759291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:20.759353 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.759355 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:06:20.759397 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:06:20.759437 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260924 05:06:20.759841 14263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc44b4fd56044edbd36ce01f8dd6d3e" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e968d05c3e745989494cda4f5ef35a2" is_pre_election: true
I20260924 05:06:20.759989 14263 raft_consensus.cc:2454] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 0.
I20260924 05:06:20.759958 14397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc44b4fd56044edbd36ce01f8dd6d3e" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4419fa25de374c7ebb218618cdc7f720" is_pre_election: true
I20260924 05:06:20.760092 14397 raft_consensus.cc:2454] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 0.
I20260924 05:06:20.760193 14467 leader_election.cc:302] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1c348e9d3340b2b56303fea2e370be, 9e968d05c3e745989494cda4f5ef35a2; no voters:
I20260924 05:06:20.760299 14599 raft_consensus.cc:2781] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:06:20.760360 14599 raft_consensus.cc:482] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:06:20.760380 14599 raft_consensus.cc:3037] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:20.761005 14599 raft_consensus.cc:504] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.761113 14599 leader_election.cc:288] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 election: Requested vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871)
I20260924 05:06:20.761245 14263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc44b4fd56044edbd36ce01f8dd6d3e" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e968d05c3e745989494cda4f5ef35a2"
I20260924 05:06:20.761258 14397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc44b4fd56044edbd36ce01f8dd6d3e" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4419fa25de374c7ebb218618cdc7f720"
I20260924 05:06:20.761314 14263 raft_consensus.cc:3037] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:20.761322 14397 raft_consensus.cc:3037] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:20.762028 14397 raft_consensus.cc:2454] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 1.
I20260924 05:06:20.762037 14263 raft_consensus.cc:2454] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 1.
I20260924 05:06:20.762187 14466 leader_election.cc:302] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4419fa25de374c7ebb218618cdc7f720, 6d1c348e9d3340b2b56303fea2e370be; no voters:
I20260924 05:06:20.762259 14599 raft_consensus.cc:2781] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:20.762387 14599 raft_consensus.cc:686] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 1 LEADER]: Becoming Leader. State: Replica: 6d1c348e9d3340b2b56303fea2e370be, State: Running, Role: LEADER
I20260924 05:06:20.762468 14599 consensus_queue.cc:237] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:20.763195 14118 catalog_manager.cc:5495] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d1c348e9d3340b2b56303fea2e370be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } health_report { overall_health: UNKNOWN } } }
I20260924 05:06:20.771822 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.769930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.98.3:44833)
0924 05:06:20.769973 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:20.769974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:20.770004 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:06:20.770031 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:20.770031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:20.770047 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:20.770095 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:20.770178 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.770625 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.771282 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:20.771454 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:20.771493 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:20.771590 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:20.771678 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:06:20.771726 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:20.771737 (+ 11us) server_negotiation.cc:299] Negotiation successful
0924 05:06:20.771751 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:06:20.773402 14263 raft_consensus.cc:1261] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1c348e9d3340b2b56303fea2e370be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:20.773411 14397 raft_consensus.cc:1261] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Refusing update from remote peer 6d1c348e9d3340b2b56303fea2e370be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:20.773741 14599 consensus_queue.cc:1035] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:20.773865 14601 consensus_queue.cc:1035] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:20.775602 14604 mvcc.cc:205] Tried to move back new op lower bound from 7332767255645487104 to 7332767255603691520. Current Snapshot: MvccSnapshot[applied={T|T < 7332767255645487104}]
I20260924 05:06:20.776157 14606 mvcc.cc:205] Tried to move back new op lower bound from 7332767255645487104 to 7332767255603691520. Current Snapshot: MvccSnapshot[applied={T|T < 7332767255645487104}]
I20260924 05:06:20.776576 14605 mvcc.cc:205] Tried to move back new op lower bound from 7332767255645487104 to 7332767255603691520. Current Snapshot: MvccSnapshot[applied={T|T < 7332767255645487104 or (T in {7332767255645487104})}]
I20260924 05:06:20.778105 14118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:20.778179 14118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:20.778810 14118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Sending DeleteTablet for 3 replicas of tablet dcc44b4fd56044edbd36ce01f8dd6d3e
I20260924 05:06:20.779106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14180
I20260924 05:06:20.779189 14243 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56364
I20260924 05:06:20.779218 14511 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60834
I20260924 05:06:20.779740 14377 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34436
I20260924 05:06:20.780138 14615 tablet_replica.cc:330] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: stopping tablet replica
I20260924 05:06:20.780244 14615 raft_consensus.cc:2229] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:20.780332 14615 raft_consensus.cc:2258] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:20.780735 14615 ts_tablet_manager.cc:1905] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:20.781875 14614 tablet_replica.cc:330] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: stopping tablet replica
I20260924 05:06:20.781989 14615 ts_tablet_manager.cc:1918] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:20.782058 14614 raft_consensus.cc:2229] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:06:20.782099 14615 log.cc:1196] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/dcc44b4fd56044edbd36ce01f8dd6d3e
I20260924 05:06:20.782220 14614 raft_consensus.cc:2258] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:20.782425 14615 ts_tablet_manager.cc:1939] T dcc44b4fd56044edbd36ce01f8dd6d3e P 4419fa25de374c7ebb218618cdc7f720: Deleting consensus metadata
I20260924 05:06:20.782714 14614 ts_tablet_manager.cc:1905] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:20.782856 14104 catalog_manager.cc:4841] TS 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561): tablet dcc44b4fd56044edbd36ce01f8dd6d3e (table test-table [id=7a618bc0816842578f021138c4dcadc4]) successfully deleted
I20260924 05:06:20.784010 14614 ts_tablet_manager.cc:1918] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:20.784068 14614 log.cc:1196] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/dcc44b4fd56044edbd36ce01f8dd6d3e
I20260924 05:06:20.784339 14614 ts_tablet_manager.cc:1939] T dcc44b4fd56044edbd36ce01f8dd6d3e P 6d1c348e9d3340b2b56303fea2e370be: Deleting consensus metadata
I20260924 05:06:20.784662 14104 catalog_manager.cc:4841] TS 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833): tablet dcc44b4fd56044edbd36ce01f8dd6d3e (table test-table [id=7a618bc0816842578f021138c4dcadc4]) successfully deleted
W20260924 05:06:20.784982 14105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32871 (error 108)
I20260924 05:06:20.785228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14314
I20260924 05:06:20.785408 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:20.785266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32871 (local address 127.0.0.1:56374)
0924 05:06:20.785347 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:20.785379 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 05:06:20.785920 14105 catalog_manager.cc:4584] TS 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcc44b4fd56044edbd36ce01f8dd6d3e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32871: connect: Connection refused (error 111)
I20260924 05:06:20.790325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14448
I20260924 05:06:20.794235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14086
I20260924 05:06:20.798493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33755
--webserver_interface=127.0.98.62
--webserver_port=36431
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:20.879240 14617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:20.879439 14617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:20.879459 14617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:20.880620 14617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:20.880663 14617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:20.880676 14617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:20.880697 14617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:20.880719 14617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:20.882242 14617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:20.882475 14617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:20.882678 14617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:20.890609 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.890590 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:20.890582 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:20.890826 14617 server_base.cc:1034] running on GCE node
I20260924 05:06:20.891072 14617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:20.891342 14617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:20.892477 14617 hybrid_clock.cc:648] HybridClock initialized: now 1790226380892459 us; error 32 us; skew 500 ppm
Sep 24 05:06:20 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:20.894858 14617 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:20.895277 14617 webserver.cc:460] Webserver started at http://127.0.98.62:36431/ using document root <none> and password file <none>
I20260924 05:06:20.895479 14617 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:20.895541 14617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:20.896829 14617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:20.897398 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:20.897536 14617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:20.897620 14617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "324ab9e684084869b2a6130548cb31e1"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "822b391c141b20489712eb10b56b301b"
server_key_iv: "03c2e8f6030585a00760e8dc0ec24310"
server_key_version: "encryptionkey@0"
I20260924 05:06:20.898049 14617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:20.907900 14617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:20.909970 14617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:20.910099 14617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:20.913812 14617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33755
I20260924 05:06:20.913859 14685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33755 every 8 connection(s)
I20260924 05:06:20.914217 14617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:20.915710 14686 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:06:20.916352 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap starting.
I20260924 05:06:20.922480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14617
I20260924 05:06:20.922737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33755
--webserver_port=35739
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:20.924412 14686 log.cc:824] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:20.926427 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:20.926678 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap complete.
I20260924 05:06:20.928836 14686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.929081 14686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324ab9e684084869b2a6130548cb31e1, State: Initialized, Role: FOLLOWER
I20260924 05:06:20.929253 14686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.929338 14686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:20.929394 14686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:20.929442 14686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:06:20.930546 14686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.930646 14686 leader_election.cc:302] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324ab9e684084869b2a6130548cb31e1; no voters:
I20260924 05:06:20.930802 14686 leader_election.cc:288] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:06:20.930873 14690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:06:20.931059 14686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:20.931066 14690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 LEADER]: Becoming Leader. State: Replica: 324ab9e684084869b2a6130548cb31e1, State: Running, Role: LEADER
I20260924 05:06:20.931217 14690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:20.931478 14690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "324ab9e684084869b2a6130548cb31e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } } }
I20260924 05:06:20.931533 14690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:20.931633 14690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324ab9e684084869b2a6130548cb31e1. Latest consensus state: current_term: 2 leader_uuid: "324ab9e684084869b2a6130548cb31e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } } }
I20260924 05:06:20.931665 14690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:20.932181 14696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:20.933066 14696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:20.933162 14696 catalog_manager.cc:1259] Loaded cluster ID: 9c6a13e021d1444fba2b8edcce9f677a
I20260924 05:06:20.933192 14696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:20.933411 14696 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 05:06:20.938458 14696 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 05:06:20.941602 14696 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:20.941843 14696 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Loaded TSK: 0
I20260924 05:06:20.942076 14696 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:06:21.004632 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:21.004835 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:21.004859 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:21.006048 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:21.006088 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:21.006112 14688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:21.007557 14688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35739
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:21.007782 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:21.007987 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:21.012485 14713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.012490 14714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.012595 14716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.012841 14688 server_base.cc:1034] running on GCE node
I20260924 05:06:21.013028 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:21.013243 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:21.014374 14688 hybrid_clock.cc:648] HybridClock initialized: now 1790226381014360 us; error 30 us; skew 500 ppm
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:21.016757 14688 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:21.017197 14688 webserver.cc:460] Webserver started at http://127.0.98.1:35739/ using document root <none> and password file <none>
I20260924 05:06:21.017410 14688 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:21.017469 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:21.018767 14688 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:21.019351 14723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.019508 14688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:21.019608 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "9e968d05c3e745989494cda4f5ef35a2"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "c24860797dc5dcd8d24ea9485404689e"
server_key_iv: "bfedb11424d6bd0c63172769dc30cf19"
server_key_version: "encryptionkey@0"
I20260924 05:06:21.020058 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:21.052605 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:21.054844 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:21.054991 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:21.055209 14688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:21.055662 14730 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:06:21.057037 14688 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:06:21.057087 14688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:06:21.057111 14688 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:06:21.057827 14688 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:06:21.057869 14688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:21.057921 14730 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap starting.
I20260924 05:06:21.065258 14688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32871
I20260924 05:06:21.065289 14837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32871 every 8 connection(s)
I20260924 05:06:21.065781 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:21.067746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14688
I20260924 05:06:21.068068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39561
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33755
--webserver_port=45739
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:21.068523 14730 log.cc:824] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:21.071044 14730 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:21.071280 14730 tablet_bootstrap.cc:492] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap complete.
I20260924 05:06:21.072240 14730 ts_tablet_manager.cc:1397] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:21.073292 14730 raft_consensus.cc:348] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.073550 14730 raft_consensus.cc:729] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e968d05c3e745989494cda4f5ef35a2, State: Initialized, Role: FOLLOWER
I20260924 05:06:21.073686 14730 consensus_queue.cc:260] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.073963 14730 ts_tablet_manager.cc:1428] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:06:21.077169 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.066611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44663 (local address 127.0.98.62:33755)
0924 05:06:21.066786 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.066793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.066830 (+ 37us) server_negotiation.cc:407] Connection header received
0924 05:06:21.067586 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.067591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.067632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.067743 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:21.068360 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.069486 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.070930 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.071398 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.071493 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.074090 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:21.074094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:21.074095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:21.074106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:21.075540 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.076007 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.076009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.076010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.076086 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.076338 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.076339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.076339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.076539 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:21.076597 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.076932 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.077031 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260924 05:06:21.077360 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.066478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:44663)
0924 05:06:21.066676 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.066727 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.067386 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.067788 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.067792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.068059 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:21.068247 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.068251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.069630 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.069631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.070842 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.070845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.071510 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.071511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.071648 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.072182 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:21.072191 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:21.073952 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:21.075618 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.075627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.075629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.075936 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.076144 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.076144 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.076145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.076280 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.076600 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:21.076606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:21.076769 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.076898 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.077065 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260924 05:06:21.078071 14838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33755
I20260924 05:06:21.078162 14838 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:21.078317 14838 heartbeater.cc:511] Master 127.0.98.62:33755 requested a full tablet report, sending...
I20260924 05:06:21.079007 14650 ts_manager.cc:194] Registered new tserver with Master: 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871)
I20260924 05:06:21.080204 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44663
I20260924 05:06:21.081275 14838 heartbeater.cc:503] Master 127.0.98.62:33755 was elected leader, sending a full tablet report...
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:06:21.086757 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.079624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.0.98.1:32871)
0924 05:06:21.079753 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.079757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.079909 (+ 152us) server_negotiation.cc:407] Connection header received
0924 05:06:21.079974 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.079977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.079990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.080044 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:21.080495 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.081160 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.082033 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.082307 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.082363 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.084349 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:21.084356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:21.084356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:21.084363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:21.085663 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.086016 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.086019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.086019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.086070 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.086228 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.086229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.086229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.086357 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:21.086424 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.086585 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.086666 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 05:06:21.086759 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.079564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32871 (local address 127.0.0.1:56386)
0924 05:06:21.079815 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.079840 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.079904 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.080053 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.080057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.080338 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:21.080402 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.080405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.081234 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.081235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.081967 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.081970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.082368 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.082372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.082454 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.082919 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:21.082923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:21.084276 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:21.085754 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.085761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.085763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.085946 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.086123 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.086124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.086124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.086180 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.086420 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:21.086426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:21.086476 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.086546 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.086656 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260924 05:06:21.087185 14772 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc44b4fd56044edbd36ce01f8dd6d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56386
I20260924 05:06:21.087760 14847 tablet_replica.cc:330] stopping tablet replica
I20260924 05:06:21.087857 14847 raft_consensus.cc:2229] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:21.087931 14847 raft_consensus.cc:2258] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:21.088227 14847 ts_tablet_manager.cc:1905] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:21.089917 14847 ts_tablet_manager.cc:1918] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:21.089982 14847 log.cc:1196] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/dcc44b4fd56044edbd36ce01f8dd6d3e
I20260924 05:06:21.090165 14847 ts_tablet_manager.cc:1939] T dcc44b4fd56044edbd36ce01f8dd6d3e P 9e968d05c3e745989494cda4f5ef35a2: Deleting consensus metadata
I20260924 05:06:21.090628 14637 catalog_manager.cc:4841] TS 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871): tablet dcc44b4fd56044edbd36ce01f8dd6d3e (table test-table [id=7a618bc0816842578f021138c4dcadc4]) successfully deleted
W20260924 05:06:21.151978 14843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:21.152179 14843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:21.152199 14843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:21.153389 14843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:21.153431 14843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:21.153458 14843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:21.154867 14843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45739
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:21.155071 14843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:21.155272 14843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:21.159710 14852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.159739 14855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.159718 14853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.159876 14843 server_base.cc:1034] running on GCE node
I20260924 05:06:21.160175 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:21.160413 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:21.161538 14843 hybrid_clock.cc:648] HybridClock initialized: now 1790226381161520 us; error 33 us; skew 500 ppm
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:21.163782 14843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:21.164158 14843 webserver.cc:460] Webserver started at http://127.0.98.2:45739/ using document root <none> and password file <none>
I20260924 05:06:21.164336 14843 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:21.164388 14843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:21.165611 14843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:21.166121 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.166241 14843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.166314 14843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "4419fa25de374c7ebb218618cdc7f720"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "d6655da95818b2fac8809b48ea1088df"
server_key_iv: "188c34843d2e5a7831ac9b3bddaa763e"
server_key_version: "encryptionkey@0"
I20260924 05:06:21.166628 14843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:21.193583 14843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:21.195699 14843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:21.195835 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:21.196050 14843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:21.196333 14843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:21.196365 14843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.196393 14843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:21.196410 14843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.201757 14843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39561
I20260924 05:06:21.201812 14975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39561 every 8 connection(s)
I20260924 05:06:21.202116 14843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:06:21.203912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14843
I20260924 05:06:21.204146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44833
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33755
--webserver_port=45527
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:21.210819 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.202727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44631 (local address 127.0.98.62:33755)
0924 05:06:21.202767 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.202769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.202977 (+ 208us) server_negotiation.cc:407] Connection header received
0924 05:06:21.203464 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.203465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.203479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.203529 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:21.204497 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.204981 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.205979 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.206173 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.206225 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.208326 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:21.208328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:21.208328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:21.208335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:21.209579 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.209966 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.209967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.209967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.210017 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.210223 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.210224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.210224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.210329 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:21.210386 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.210636 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.210726 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:06:21.211002 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.202723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:44631)
0924 05:06:21.202864 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.202900 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.203366 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.203598 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.203602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.204167 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:21.204411 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.204417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.205058 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.205059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.205908 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.205910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.206236 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.206237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.206357 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.206818 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:21.206826 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:21.208244 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:21.209646 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.209652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.209654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.209904 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.210068 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.210069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.210069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.210171 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.210381 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:21.210386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:21.210525 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.210609 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.210749 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260924 05:06:21.211509 14976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33755
I20260924 05:06:21.211596 14976 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:21.211756 14976 heartbeater.cc:511] Master 127.0.98.62:33755 requested a full tablet report, sending...
I20260924 05:06:21.212127 14650 ts_manager.cc:194] Registered new tserver with Master: 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561)
I20260924 05:06:21.212630 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44631
W20260924 05:06:21.284219 14979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:21.284410 14979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:21.284440 14979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:21.285539 14979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:21.285579 14979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:21.285614 14979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:21.287029 14979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45527
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:21.287254 14979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:21.287451 14979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:21.292135 14984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.292130 14985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.292135 14987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.292331 14979 server_base.cc:1034] running on GCE node
I20260924 05:06:21.292613 14979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:21.292866 14979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:21.293995 14979 hybrid_clock.cc:648] HybridClock initialized: now 1790226381293982 us; error 26 us; skew 500 ppm
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:21.296319 14979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:21.296726 14979 webserver.cc:460] Webserver started at http://127.0.98.3:45527/ using document root <none> and password file <none>
I20260924 05:06:21.296933 14979 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:21.296989 14979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:21.298280 14979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:21.298791 14994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.298926 14979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:21.298995 14979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "6d1c348e9d3340b2b56303fea2e370be"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "6c397102699dfb89fea3e45f25dbc6b5"
server_key_iv: "221e2f3ac30b5df10447971c7d5c07ed"
server_key_version: "encryptionkey@0"
I20260924 05:06:21.299310 14979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:21.305408 14979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:21.307287 14979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:21.307405 14979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:21.307588 14979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:21.307838 14979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:21.307869 14979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.307888 14979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:21.307911 14979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.313186 14979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44833
I20260924 05:06:21.313241 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44833 every 8 connection(s)
I20260924 05:06:21.313509 14979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:06:21.318620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 14979
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:21.321887 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.314149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54273 (local address 127.0.98.62:33755)
0924 05:06:21.314191 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.314193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.314390 (+ 197us) server_negotiation.cc:407] Connection header received
0924 05:06:21.314823 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.314824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.314832 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.314881 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:21.315684 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.316158 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.317092 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.317260 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.317300 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.319371 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:21.319373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:21.319373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:21.319380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:21.320689 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.321038 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.321039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.321039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.321094 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.321304 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.321305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.321305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.321407 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:21.321454 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.321757 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.321819 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:06:21.322118 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.314130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:54273)
0924 05:06:21.314267 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.314306 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.314750 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.314931 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.314934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.315445 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:21.315619 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.315623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.316217 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.316218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.317033 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.317035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.317313 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.317314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.317402 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.317869 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:21.317877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:21.319283 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:21.320762 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.320768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.320770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.320977 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.321146 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:21.321147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:21.321147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:21.321248 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:21.321458 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:21.321463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:21.321615 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.321696 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.321894 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260924 05:06:21.322573 15108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33755
I20260924 05:06:21.322664 15108 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:21.322862 15108 heartbeater.cc:511] Master 127.0.98.62:33755 requested a full tablet report, sending...
I20260924 05:06:21.323236 14650 ts_manager.cc:194] Registered new tserver with Master: 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833)
I20260924 05:06:21.323683 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54273
I20260924 05:06:21.325047 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.318950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.0.98.62:33755)
0924 05:06:21.319057 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.319059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.319094 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:06:21.319143 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.319143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.319154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.319211 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:21.319809 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.320394 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.321244 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.321484 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.321543 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.322731 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:21.322732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:21.322733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:21.322739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:21.324057 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.324375 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.324376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.324377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.324434 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:21.324590 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:21.324590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:21.324591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:21.324692 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:21.324848 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.324883 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.324957 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
I20260924 05:06:21.325685 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:06:21.327934 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.325987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.0.98.62:33755)
0924 05:06:21.326038 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.326040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.326111 (+ 71us) server_negotiation.cc:407] Connection header received
0924 05:06:21.326166 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.326167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.326193 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.326244 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:21.326316 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.326747 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.327401 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.327571 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.327611 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.327716 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:21.327800 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:06:21.327853 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.327865 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.327880 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:06:21.328173 14650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:06:21.329695 14650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:06:21.331678 14772 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9170211416475398881576e2b225d1 (DEFAULT_TABLE table=test-table [id=35c614697521483d8be50450cb1b3ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:21.331866 14772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9170211416475398881576e2b225d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:21.333146 14730 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap starting.
I20260924 05:06:21.333446 14730 tablet_bootstrap.cc:654] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:21.334003 14730 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: No bootstrap required, opened a new log
I20260924 05:06:21.334049 14730 ts_tablet_manager.cc:1397] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:21.334390 14730 raft_consensus.cc:348] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.334470 14730 raft_consensus.cc:374] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:21.334510 14730 raft_consensus.cc:729] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e968d05c3e745989494cda4f5ef35a2, State: Initialized, Role: FOLLOWER
I20260924 05:06:21.334556 14730 consensus_queue.cc:260] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.334638 14730 ts_tablet_manager.cc:1428] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:21.336908 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.331562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.98.3:44833)
0924 05:06:21.331727 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.331732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.333541 (+ 1809us) server_negotiation.cc:407] Connection header received
0924 05:06:21.333602 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.333604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.333620 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.333842 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:21.333915 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.334558 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.335987 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.336544 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.336607 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.336667 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.336787 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.336824 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260924 05:06:21.336913 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.331484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.0.98.2:39561)
0924 05:06:21.333209 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.333215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.333225 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:06:21.333266 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.333268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.333286 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.333370 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:21.333517 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.334579 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.336018 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.336548 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.336611 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.336667 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.336788 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.336828 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":83,"threads_started":1}
I20260924 05:06:21.336923 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.331382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39561 (local address 127.0.0.1:34438)
0924 05:06:21.331424 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.331435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.331469 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.333378 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.333379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.333381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:21.333429 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.333431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.334712 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.334713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.335942 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.335945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.336626 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.336627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.336699 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.336720 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.336769 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.336818 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":71}
I20260924 05:06:21.336925 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.331478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44833 (local address 127.0.0.1:60842)
0924 05:06:21.333480 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.333491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.333538 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.333706 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.333707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.333731 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:21.333769 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.333771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.334759 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.334759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.335917 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.335919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.336619 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.336620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.336696 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.336719 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.336767 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.336821 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260924 05:06:21.337621 14910 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9170211416475398881576e2b225d1 (DEFAULT_TABLE table=test-table [id=35c614697521483d8be50450cb1b3ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:21.337848 15042 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9170211416475398881576e2b225d1 (DEFAULT_TABLE table=test-table [id=35c614697521483d8be50450cb1b3ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:21.337965 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9170211416475398881576e2b225d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:21.338109 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9170211416475398881576e2b225d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:21.340600 15119 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Bootstrap starting.
I20260924 05:06:21.341132 15119 tablet_bootstrap.cc:654] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:21.341557 15119 log.cc:824] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:21.341845 15118 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Bootstrap starting.
I20260924 05:06:21.342294 15119 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: No bootstrap required, opened a new log
I20260924 05:06:21.342365 15119 ts_tablet_manager.cc:1397] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:06:21.342459 15118 tablet_bootstrap.cc:654] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:21.342952 15118 log.cc:824] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:21.343638 15118 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: No bootstrap required, opened a new log
I20260924 05:06:21.343705 15118 ts_tablet_manager.cc:1397] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:21.344175 15119 raft_consensus.cc:348] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.344350 15119 raft_consensus.cc:374] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:21.344434 15119 raft_consensus.cc:729] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1c348e9d3340b2b56303fea2e370be, State: Initialized, Role: FOLLOWER
I20260924 05:06:21.344594 15119 consensus_queue.cc:260] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.344871 15119 ts_tablet_manager.cc:1428] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:06:21.344895 15108 heartbeater.cc:503] Master 127.0.98.62:33755 was elected leader, sending a full tablet report...
I20260924 05:06:21.345371 15118 raft_consensus.cc:348] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.345496 15118 raft_consensus.cc:374] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:21.345517 15118 raft_consensus.cc:729] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4419fa25de374c7ebb218618cdc7f720, State: Initialized, Role: FOLLOWER
I20260924 05:06:21.345641 15118 consensus_queue.cc:260] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.345913 15118 ts_tablet_manager.cc:1428] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:06:21.345933 14976 heartbeater.cc:503] Master 127.0.98.62:33755 was elected leader, sending a full tablet report...
W20260924 05:06:21.452642 14977 tablet.cc:2369] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:06:21.564052 15109 tablet.cc:2369] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:06:21.566474 14839 tablet.cc:2369] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:06:21.630375 15122 raft_consensus.cc:482] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:21.630514 15122 raft_consensus.cc:504] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.630833 15122 leader_election.cc:288] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871)
I20260924 05:06:21.633849 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.631166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53949 (local address 127.0.98.2:39561)
0924 05:06:21.631207 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.631209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.631219 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:06:21.631249 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.631250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.631261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.631334 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:21.631490 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.632093 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.633107 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.633487 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.633538 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.633547 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.633736 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.633776 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:06:21.633852 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.631115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36379 (local address 127.0.98.1:32871)
0924 05:06:21.631303 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.631305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.631315 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:06:21.631374 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.631375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.631403 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.631483 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:21.631627 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.632148 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.633107 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.633532 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.633602 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.633692 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.633737 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.633775 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
I20260924 05:06:21.633864 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.631060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39561 (local address 127.0.98.3:53949)
0924 05:06:21.631122 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.631133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.631180 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.631349 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.631350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.631352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:21.631426 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.631428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.632159 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.632160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.633014 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.633017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.633564 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.633565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.633646 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.633649 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.633699 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.633760 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":32}
I20260924 05:06:21.633870 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.631060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32871 (local address 127.0.98.3:36379)
0924 05:06:21.631261 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.631271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:21.631319 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:21.631484 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:21.631485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:21.631486 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:21.631546 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.631548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.632232 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.632233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.633020 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:21.633023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.633617 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.633618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:21.633668 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.633670 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:06:21.633726 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:06:21.633764 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":39,"thread_start_us":108,"threads_started":1}
I20260924 05:06:21.634202 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9170211416475398881576e2b225d1" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4419fa25de374c7ebb218618cdc7f720" is_pre_election: true
I20260924 05:06:21.634198 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9170211416475398881576e2b225d1" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e968d05c3e745989494cda4f5ef35a2" is_pre_election: true
I20260924 05:06:21.634341 14792 raft_consensus.cc:2454] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 0.
I20260924 05:06:21.634346 14930 raft_consensus.cc:2454] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 0.
I20260924 05:06:21.634565 14997 leader_election.cc:302] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4419fa25de374c7ebb218618cdc7f720, 6d1c348e9d3340b2b56303fea2e370be; no voters:
I20260924 05:06:21.634729 15122 raft_consensus.cc:2781] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:06:21.634768 15122 raft_consensus.cc:482] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:06:21.634781 15122 raft_consensus.cc:3037] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:21.635419 15122 raft_consensus.cc:504] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.635533 15122 leader_election.cc:288] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 election: Requested vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871)
I20260924 05:06:21.635689 14792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9170211416475398881576e2b225d1" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e968d05c3e745989494cda4f5ef35a2"
I20260924 05:06:21.635705 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9170211416475398881576e2b225d1" candidate_uuid: "6d1c348e9d3340b2b56303fea2e370be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4419fa25de374c7ebb218618cdc7f720"
I20260924 05:06:21.635766 14792 raft_consensus.cc:3037] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:21.635773 14930 raft_consensus.cc:3037] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:21.636364 14792 raft_consensus.cc:2454] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 1.
I20260924 05:06:21.636498 14930 raft_consensus.cc:2454] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d1c348e9d3340b2b56303fea2e370be in term 1.
I20260924 05:06:21.636535 14998 leader_election.cc:302] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d1c348e9d3340b2b56303fea2e370be, 9e968d05c3e745989494cda4f5ef35a2; no voters:
I20260924 05:06:21.636615 15122 raft_consensus.cc:2781] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:21.636765 15122 raft_consensus.cc:686] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 1 LEADER]: Becoming Leader. State: Replica: 6d1c348e9d3340b2b56303fea2e370be, State: Running, Role: LEADER
I20260924 05:06:21.636859 15122 consensus_queue.cc:237] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:21.637590 14650 catalog_manager.cc:5495] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d1c348e9d3340b2b56303fea2e370be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } health_report { overall_health: UNKNOWN } } }
I20260924 05:06:21.686925 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.684872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.0.98.3:44833)
0924 05:06:21.684922 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:21.684923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:21.684931 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:21.684966 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:21.684967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:21.684986 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:21.685059 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:21.685150 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.685633 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.686329 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:21.686524 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:21.686572 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:21.686675 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:21.686780 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:06:21.686836 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:21.686849 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 05:06:21.686865 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:06:21.688747 14930 raft_consensus.cc:1261] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Refusing update from remote peer 6d1c348e9d3340b2b56303fea2e370be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:21.688746 14792 raft_consensus.cc:1261] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Refusing update from remote peer 6d1c348e9d3340b2b56303fea2e370be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:21.689123 15126 consensus_queue.cc:1035] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:21.689260 15122 consensus_queue.cc:1035] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:21.691468 15128 mvcc.cc:205] Tried to move back new op lower bound from 7332767259394228224 to 7332767259185061888. Current Snapshot: MvccSnapshot[applied={T|T < 7332767259394228224}]
I20260924 05:06:21.691587 15130 mvcc.cc:205] Tried to move back new op lower bound from 7332767259394228224 to 7332767259185061888. Current Snapshot: MvccSnapshot[applied={T|T < 7332767259394228224}]
I20260924 05:06:21.691985 15129 mvcc.cc:205] Tried to move back new op lower bound from 7332767259394228224 to 7332767259185061888. Current Snapshot: MvccSnapshot[applied={T|T < 7332767259394228224}]
I20260924 05:06:21.693851 14649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:21.693943 14649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:21.694577 14649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Sending DeleteTablet for 3 replicas of tablet 6c9170211416475398881576e2b225d1
I20260924 05:06:21.694846 14910 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9170211416475398881576e2b225d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34438
I20260924 05:06:21.694881 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9170211416475398881576e2b225d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60842
I20260924 05:06:21.695046 14772 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9170211416475398881576e2b225d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56386
I20260924 05:06:21.695139 15139 tablet_replica.cc:330] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: stopping tablet replica
I20260924 05:06:21.695253 15139 raft_consensus.cc:2229] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:06:21.695274 15140 tablet_replica.cc:330] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: stopping tablet replica
I20260924 05:06:21.695333 15140 raft_consensus.cc:2229] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:21.695356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14688
I20260924 05:06:21.695395 15140 raft_consensus.cc:2258] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:21.696600 15138 tablet_replica.cc:330] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: stopping tablet replica
I20260924 05:06:21.696692 15138 raft_consensus.cc:2229] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:21.696789 15138 raft_consensus.cc:2258] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:21.697204 15138 ts_tablet_manager.cc:1905] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:21.698688 15138 ts_tablet_manager.cc:1918] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:21.698755 15138 log.cc:1196] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/6c9170211416475398881576e2b225d1
I20260924 05:06:21.698971 15138 ts_tablet_manager.cc:1939] T 6c9170211416475398881576e2b225d1 P 4419fa25de374c7ebb218618cdc7f720: Deleting consensus metadata
I20260924 05:06:21.699360 14636 catalog_manager.cc:4841] TS 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561): tablet 6c9170211416475398881576e2b225d1 (table test-table [id=35c614697521483d8be50450cb1b3ae8]) successfully deleted
W20260924 05:06:21.701433 14637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32871 (error 108)
I20260924 05:06:21.701738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14843
I20260924 05:06:21.701758 15139 raft_consensus.cc:2258] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:21.702636 15139 ts_tablet_manager.cc:1905] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:21.703421 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:21.703309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32871 (local address 127.0.0.1:56402)
0924 05:06:21.703346 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:21.703386 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 05:06:21.704162 15139 ts_tablet_manager.cc:1918] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:21.704219 15139 log.cc:1196] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/6c9170211416475398881576e2b225d1
I20260924 05:06:21.704483 15139 ts_tablet_manager.cc:1939] T 6c9170211416475398881576e2b225d1 P 6d1c348e9d3340b2b56303fea2e370be: Deleting consensus metadata
W20260924 05:06:21.704427 14637 catalog_manager.cc:4584] TS 9e968d05c3e745989494cda4f5ef35a2 (127.0.98.1:32871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c9170211416475398881576e2b225d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32871: connect: Connection refused (error 111)
I20260924 05:06:21.705000 14636 catalog_manager.cc:4841] TS 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833): tablet 6c9170211416475398881576e2b225d1 (table test-table [id=35c614697521483d8be50450cb1b3ae8]) successfully deleted
I20260924 05:06:21.706537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14979
I20260924 05:06:21.712244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 14617
I20260924 05:06:21.718048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33755
--webserver_interface=127.0.98.62
--webserver_port=36431
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:21.797605 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:21.797849 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:21.797873 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:21.798955 15142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:21.798990 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:21.799002 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:21.799013 15142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:21.799031 15142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:21.800436 15142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:21.800630 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:21.800802 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:21.805198 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.805220 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.805389 15142 server_base.cc:1034] running on GCE node
W20260924 05:06:21.805191 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.805696 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:21.806016 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:21.807183 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790226381807170 us; error 26 us; skew 500 ppm
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:21.813131 15142 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:21.813582 15142 webserver.cc:460] Webserver started at http://127.0.98.62:36431/ using document root <none> and password file <none>
I20260924 05:06:21.813802 15142 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:21.813854 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:21.815096 15142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:06:21.815663 15157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.815801 15142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:06:21.815881 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "324ab9e684084869b2a6130548cb31e1"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "822b391c141b20489712eb10b56b301b"
server_key_iv: "03c2e8f6030585a00760e8dc0ec24310"
server_key_version: "encryptionkey@0"
I20260924 05:06:21.816309 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:21.849207 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:21.850955 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:21.851089 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:21.854890 15142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33755
I20260924 05:06:21.854933 15209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33755 every 8 connection(s)
I20260924 05:06:21.855281 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:21.856750 15210 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:06:21.857335 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap starting.
I20260924 05:06:21.862283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15142
I20260924 05:06:21.862592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33755
--webserver_port=35739
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:21.865656 15210 log.cc:824] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:21.867887 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:21.868162 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1: Bootstrap complete.
I20260924 05:06:21.870312 15210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:21.870529 15210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324ab9e684084869b2a6130548cb31e1, State: Initialized, Role: FOLLOWER
I20260924 05:06:21.870671 15210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:21.870728 15210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:21.870767 15210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:21.870812 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:06:21.871753 15210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:21.871837 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324ab9e684084869b2a6130548cb31e1; no voters:
I20260924 05:06:21.872005 15210 leader_election.cc:288] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 05:06:21.872068 15214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:06:21.872210 15214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [term 3 LEADER]: Becoming Leader. State: Replica: 324ab9e684084869b2a6130548cb31e1, State: Running, Role: LEADER
I20260924 05:06:21.872259 15210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:21.872301 15214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } }
I20260924 05:06:21.872604 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324ab9e684084869b2a6130548cb31e1. Latest consensus state: current_term: 3 leader_uuid: "324ab9e684084869b2a6130548cb31e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } } }
I20260924 05:06:21.872679 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:21.872615 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "324ab9e684084869b2a6130548cb31e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324ab9e684084869b2a6130548cb31e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33755 } } }
I20260924 05:06:21.872852 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324ab9e684084869b2a6130548cb31e1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:21.873178 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:21.873993 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:21.874101 15220 catalog_manager.cc:1259] Loaded cluster ID: 9c6a13e021d1444fba2b8edcce9f677a
I20260924 05:06:21.874300 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 05:06:21.874509 15220 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd69317b9ad google::LogMessage::Fail() at ??:0
@ 0x7fd69317e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd69317b4de google::LogMessage::Flush() at ??:0
@ 0x7fd69317ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd69b09c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd69b0a58b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd6950490d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd69503e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd695ea26db start_thread at ??:0
@ 0x7fd691c7961f clone at ??:0
W20260924 05:06:21.950480 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:21.950690 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:21.950724 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:21.951860 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:21.951901 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:21.951936 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:21.953368 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35739
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:21.953608 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:21.953850 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:21.958520 15247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:21.958550 15249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.958654 15212 server_base.cc:1034] running on GCE node
W20260924 05:06:21.958532 15246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:21.958935 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:21.959165 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:21.960314 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790226381960299 us; error 31 us; skew 500 ppm
Sep 24 05:06:21 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:21.962630 15212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:21.963054 15212 webserver.cc:460] Webserver started at http://127.0.98.1:35739/ using document root <none> and password file <none>
I20260924 05:06:21.963253 15212 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:21.963301 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:21.964596 15212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:06:21.965227 15256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:21.965384 15212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:06:21.965481 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "9e968d05c3e745989494cda4f5ef35a2"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "c24860797dc5dcd8d24ea9485404689e"
server_key_iv: "bfedb11424d6bd0c63172769dc30cf19"
server_key_version: "encryptionkey@0"
I20260924 05:06:21.965945 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:21.988955 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:21.991530 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:21.991700 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:21.991981 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:21.992502 15263 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:06:21.993980 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:06:21.994040 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:06:21.994072 15212 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:06:21.994958 15212 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:06:21.995002 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:21.995059 15263 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap starting.
I20260924 05:06:22.001885 15212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32871
I20260924 05:06:22.002054 15370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32871 every 8 connection(s)
I20260924 05:06:22.002332 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:22.004736 15263 log.cc:824] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:22.006752 15263 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:22.006976 15263 tablet_bootstrap.cc:492] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Bootstrap complete.
I20260924 05:06:22.007730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15212
I20260924 05:06:22.007751 15263 ts_tablet_manager.cc:1397] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260924 05:06:22.008021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39561
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33755
--webserver_port=45739
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:22.009027 15263 raft_consensus.cc:348] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:22.009336 15263 raft_consensus.cc:729] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e968d05c3e745989494cda4f5ef35a2, State: Initialized, Role: FOLLOWER
I20260924 05:06:22.009507 15263 consensus_queue.cc:260] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:22.009773 15263 ts_tablet_manager.cc:1428] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260924 05:06:22.068289 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:06:22.003120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:43709)
0924 05:06:22.003335 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.003388 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:22.004047 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:22.068080 (+ 64033us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
W20260924 05:06:22.068563 15371 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33755 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33755: Client connection negotiation failed: client connection to 127.0.98.62:33755: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:06:22.068878 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.068770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:56969)
0924 05:06:22.068818 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.068857 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:06:22.069190 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.069108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:33237)
0924 05:06:22.069147 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.069171 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:06:22.069300 15371 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:06:22.088439 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:22.088646 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:22.088670 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:22.089854 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:22.089900 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:22.089931 15375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:22.091375 15375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45739
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:22.091600 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:22.091802 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:22.096300 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:22.096400 15375 server_base.cc:1034] running on GCE node
W20260924 05:06:22.096321 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:22.096305 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:22.096753 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:22.096966 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:22.098097 15375 hybrid_clock.cc:648] HybridClock initialized: now 1790226382098081 us; error 27 us; skew 500 ppm
Sep 24 05:06:22 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:22.100368 15375 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:22.100759 15375 webserver.cc:460] Webserver started at http://127.0.98.2:45739/ using document root <none> and password file <none>
I20260924 05:06:22.100943 15375 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:22.100982 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:22.102207 15375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:22.102701 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.102839 15375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.102914 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "4419fa25de374c7ebb218618cdc7f720"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "d6655da95818b2fac8809b48ea1088df"
server_key_iv: "188c34843d2e5a7831ac9b3bddaa763e"
server_key_version: "encryptionkey@0"
I20260924 05:06:22.103216 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:22.130312 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:22.132090 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:22.132205 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:22.132390 15375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:22.132653 15375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:22.132682 15375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.132704 15375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:22.132718 15375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.138317 15375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39561
I20260924 05:06:22.138374 15504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39561 every 8 connection(s)
I20260924 05:06:22.138703 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:06:22.139714 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.139341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:36761)
0924 05:06:22.139496 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.139569 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
W20260924 05:06:22.140081 15505 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33755 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33755: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
I20260924 05:06:22.140343 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.140266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:55533)
0924 05:06:22.140306 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.140327 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 05:06:22.140583 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.140517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:44967)
0924 05:06:22.140549 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.140570 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 05:06:22.140666 15505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:06:22.143213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15375
I20260924 05:06:22.143461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44833
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33755
--webserver_port=45527
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:22.222687 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:22.222910 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:22.222944 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:22.224089 15508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:22.224133 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:22.224166 15508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:22.225692 15508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45527
--tserver_master_addrs=127.0.98.62:33755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:22.225960 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:22.226172 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:06:22.231019 15508 server_base.cc:1034] running on GCE node
W20260924 05:06:22.230986 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:22.230998 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:22.230980 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:22.231334 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:22.231570 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:22.232705 15508 hybrid_clock.cc:648] HybridClock initialized: now 1790226382232697 us; error 26 us; skew 500 ppm
Sep 24 05:06:22 dist-test-slave-m01t krb5kdc[14073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:22.235160 15508 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:22.235597 15508 webserver.cc:460] Webserver started at http://127.0.98.3:45527/ using document root <none> and password file <none>
I20260924 05:06:22.235800 15508 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:22.235849 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:22.237114 15508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:06:22.237653 15523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.237809 15508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:06:22.237895 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "6d1c348e9d3340b2b56303fea2e370be"
format_stamp: "Formatted at 2026-09-24 05:06:20 on dist-test-slave-m01t"
server_key: "6c397102699dfb89fea3e45f25dbc6b5"
server_key_iv: "221e2f3ac30b5df10447971c7d5c07ed"
server_key_version: "encryptionkey@0"
I20260924 05:06:22.238240 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:22.258767 15508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:22.260926 15508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:22.261063 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:22.261281 15508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:22.261574 15508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:22.261606 15508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.261636 15508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:22.261657 15508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:22.267027 15508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44833
I20260924 05:06:22.267079 15636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44833 every 8 connection(s)
I20260924 05:06:22.267408 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:06:22.268092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15508
I20260924 05:06:22.268431 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.267998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:45155)
0924 05:06:22.268155 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.268233 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260924 05:06:22.268790 15637 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33755 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33755: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
I20260924 05:06:22.269065 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.268952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:37435)
0924 05:06:22.269009 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.269039 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:06:22.269299 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:22.269240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:41905)
0924 05:06:22.269270 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:22.269287 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 05:06:22.269394 15637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:06:23.069942 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.069631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:52529)
0924 05:06:23.069804 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:23.069849 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260924 05:06:23.141155 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.140958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:45457)
0924 05:06:23.141066 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:23.141105 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260924 05:06:23.269912 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.269695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:42197)
0924 05:06:23.269826 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:23.269858 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 05:06:23.646292 15644 raft_consensus.cc:482] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:23.646411 15644 raft_consensus.cc:504] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:23.646675 15644 leader_election.cc:288] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833)
Sep 24 05:06:23 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 05:06:23 dist-test-slave-m01t krb5kdc[14073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 05:06:23.656735 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.647068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41269 (local address 127.0.98.3:44833)
0924 05:06:23.647177 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:23.647187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:23.647220 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:06:23.647639 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:23.647641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:23.647677 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:23.647806 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:23.648556 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.649425 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.650481 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.650874 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.650971 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:23.653411 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:23.653415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:23.653416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:23.653425 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:23.655090 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655610 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:23.655613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:23.655614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:23.655701 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655935 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:23.655935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:23.655936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:23.656165 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:23.656230 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:23.656503 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 05:06:23.656619 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
I20260924 05:06:23.656731 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.646918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46121 (local address 127.0.98.2:39561)
0924 05:06:23.647125 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:23.647138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:23.647162 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:06:23.647612 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:23.647614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:23.647656 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:23.647785 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:23.648600 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.649433 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.650484 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.650879 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.650977 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:23.653028 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:23.653031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:23.653031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:23.653041 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:23.654939 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655611 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:23.655614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:23.655614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:23.655702 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655933 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:23.655934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:23.655935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:23.656164 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:23.656246 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:23.656504 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 05:06:23.656621 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":53,"threads_started":1}
I20260924 05:06:23.656725 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.646828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39561 (local address 127.0.98.1:46121)
0924 05:06:23.646916 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:23.646927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:23.646982 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:23.647844 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:23.647847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:23.648203 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:23.648414 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:23.648420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.649539 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.649540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:23.650404 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:23.650407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.650957 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.650958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:23.651058 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:23.651558 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:23.651566 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:23.652929 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:23.655172 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:23.655178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:23.655536 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:23.655768 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:23.655769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:23.655877 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:23.656240 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:23.656241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:23.656359 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 05:06:23.656445 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:06:23.656631 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":62,"thread_start_us":44,"threads_started":1}
I20260924 05:06:23.656982 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:23.646975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44833 (local address 127.0.98.1:41269)
0924 05:06:23.647048 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:23.647056 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:23.647094 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:23.647845 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:23.647848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:23.648202 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:23.648426 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:23.648429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.649518 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.649519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:23.650403 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:23.650406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:23.650952 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:23.650952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:23.651056 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:23.651599 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:23.651603 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:23.653309 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:23.655164 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:23.655173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:23.655536 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:23.655761 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:23.655762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:23.655762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:23.655878 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:23.656229 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:23.656235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:23.656359 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 05:06:23.656445 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:06:23.656711 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":161,"thread_start_us":34,"threads_started":1}
W20260924 05:06:23.657527 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
W20260924 05:06:23.657732 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
I20260924 05:06:23.657787 15259 leader_election.cc:302] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e968d05c3e745989494cda4f5ef35a2; no voters: 4419fa25de374c7ebb218618cdc7f720, 6d1c348e9d3340b2b56303fea2e370be
I20260924 05:06:23.657907 15644 raft_consensus.cc:2726] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:06:24.070438 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:24.070318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:43299)
0924 05:06:24.070366 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:24.070398 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:06:24.141824 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:24.141571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:47947)
0924 05:06:24.141683 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:24.141775 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260924 05:06:24.270462 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:24.270281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:57275)
0924 05:06:24.270389 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:24.270426 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260924 05:06:25.071069 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:25.070822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:41267)
0924 05:06:25.070939 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:25.070990 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260924 05:06:25.142400 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:25.142201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:43849)
0924 05:06:25.142318 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:25.142360 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 05:06:25.271049 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:25.270861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:39099)
0924 05:06:25.270968 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:25.271006 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260924 05:06:25.882102 15654 raft_consensus.cc:482] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:25.882215 15654 raft_consensus.cc:504] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:25.882372 15654 leader_election.cc:288] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833)
W20260924 05:06:25.882786 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
W20260924 05:06:25.882869 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
I20260924 05:06:25.882917 15259 leader_election.cc:302] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e968d05c3e745989494cda4f5ef35a2; no voters: 4419fa25de374c7ebb218618cdc7f720, 6d1c348e9d3340b2b56303fea2e370be
I20260924 05:06:25.882999 15654 raft_consensus.cc:2726] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:06:26.071713 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:26.071515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:50685)
0924 05:06:26.071633 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:26.071666 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 05:06:26.143033 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:26.142807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:50435)
0924 05:06:26.142927 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:26.142970 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 05:06:26.271677 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:26.271448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:40841)
0924 05:06:26.271576 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:26.271620 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260924 05:06:27.072314 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:27.072113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:51057)
0924 05:06:27.072233 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:27.072267 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 05:06:27.143735 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:27.143504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:37647)
0924 05:06:27.143649 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:27.143683 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260924 05:06:27.272270 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:27.272081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:48505)
0924 05:06:27.272193 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:27.272229 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 05:06:27.987675 15661 raft_consensus.cc:482] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:27.987771 15661 raft_consensus.cc:504] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:27.987908 15661 leader_election.cc:288] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833)
W20260924 05:06:27.988297 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
W20260924 05:06:27.988371 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
I20260924 05:06:27.988392 15259 leader_election.cc:302] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e968d05c3e745989494cda4f5ef35a2; no voters: 4419fa25de374c7ebb218618cdc7f720, 6d1c348e9d3340b2b56303fea2e370be
I20260924 05:06:27.988458 15661 raft_consensus.cc:2726] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:06:28.072950 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:28.072748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:35125)
0924 05:06:28.072869 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:28.072904 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260924 05:06:28.144339 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:28.144122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:52523)
0924 05:06:28.144246 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:28.144279 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260924 05:06:28.272946 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:28.272683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:46351)
0924 05:06:28.272831 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:28.272892 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260924 05:06:29.073611 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:29.073395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:60593)
0924 05:06:29.073523 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:29.073563 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260924 05:06:29.144940 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:29.144726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:49335)
0924 05:06:29.144852 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:29.144893 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 05:06:29.273548 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:29.273354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:44947)
0924 05:06:29.273467 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:29.273508 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260924 05:06:30.074265 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:30.074033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:46447)
0924 05:06:30.074175 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:30.074215 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260924 05:06:30.145577 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:30.145376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:60539)
0924 05:06:30.145496 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:30.145535 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 05:06:30.274139 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:30.273965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:39107)
0924 05:06:30.274067 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:30.274097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260924 05:06:31.074862 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:31.074663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:33145)
0924 05:06:31.074774 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:31.074808 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260924 05:06:31.146189 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:31.145979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:41905)
0924 05:06:31.146099 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:31.146142 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 05:06:31.274731 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:31.274521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:47179)
0924 05:06:31.274647 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:31.274689 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260924 05:06:32.075488 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:32.075280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:54929)
0924 05:06:32.075403 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:32.075442 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260924 05:06:32.146797 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:32.146571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:49897)
0924 05:06:32.146705 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:32.146743 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260924 05:06:32.275347 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:32.275126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:36247)
0924 05:06:32.275259 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:32.275304 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 05:06:32.992568 15677 raft_consensus.cc:482] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:32.992661 15677 raft_consensus.cc:504] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4419fa25de374c7ebb218618cdc7f720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39561 } } peers { permanent_uuid: "6d1c348e9d3340b2b56303fea2e370be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44833 } } peers { permanent_uuid: "9e968d05c3e745989494cda4f5ef35a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32871 } }
I20260924 05:06:32.992785 15677 leader_election.cc:288] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561), 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833)
W20260924 05:06:32.993175 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d1c348e9d3340b2b56303fea2e370be (127.0.98.3:44833): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
W20260924 05:06:32.993247 15259 leader_election.cc:341] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4419fa25de374c7ebb218618cdc7f720 (127.0.98.2:39561): Not found: Tablet not found: 6c9170211416475398881576e2b225d1
I20260924 05:06:32.993268 15259 leader_election.cc:302] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e968d05c3e745989494cda4f5ef35a2; no voters: 4419fa25de374c7ebb218618cdc7f720, 6d1c348e9d3340b2b56303fea2e370be
I20260924 05:06:32.993350 15677 raft_consensus.cc:2726] T 6c9170211416475398881576e2b225d1 P 9e968d05c3e745989494cda4f5ef35a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:06:33.076152 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:33.075952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:40495)
0924 05:06:33.076068 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:33.076106 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260924 05:06:33.147420 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:33.147213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:50065)
0924 05:06:33.147333 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:33.147371 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260924 05:06:33.275978 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:33.275772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:50515)
0924 05:06:33.275897 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:33.275934 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260924 05:06:34.076855 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:34.076637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:40795)
0924 05:06:34.076770 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:34.076809 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260924 05:06:34.148025 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:34.147832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:40333)
0924 05:06:34.147944 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:34.147979 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260924 05:06:34.276585 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:34.276388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:60785)
0924 05:06:34.276510 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:34.276539 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260924 05:06:35.077482 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:35.077287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:45533)
0924 05:06:35.077401 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:35.077437 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260924 05:06:35.148558 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:35.148367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:50781)
0924 05:06:35.148480 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:35.148513 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 05:06:35.277149 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:35.276955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:38571)
0924 05:06:35.277063 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:35.277104 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260924 05:06:36.078099 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:36.077904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:58643)
0924 05:06:36.078017 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:36.078053 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260924 05:06:36.149189 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:36.148983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:49065)
0924 05:06:36.149107 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:36.149142 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260924 05:06:36.277694 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:36.277505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.3:35357)
0924 05:06:36.277617 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:36.277653 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260924 05:06:37.078751 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:37.078538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.1:51375)
0924 05:06:37.078666 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:37.078703 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260924 05:06:37.149816 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:37.149616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33755 (local address 127.0.98.2:56781)
0924 05:06:37.149742 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:37.149773 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260924 05:06:37.269347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 15212
I20260924 05:06:37.273464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 15375
I20260924 05:06:37.277053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 15508
2026-09-24T05:06:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18207 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:06:37 dist-test-slave-m01t krb5kdc[15694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:06:37 dist-test-slave-m01t krb5kdc[15694](info): set up 2 sockets
Sep 24 05:06:37 dist-test-slave-m01t krb5kdc[15694](info): commencing operation
krb5kdc: starting...
W20260924 05:06:38.213411 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:06:38.320130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:38.396934 15707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:38.397128 15707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:38.397156 15707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:38.398262 15707 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:06:38.398298 15707 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:06:38.398314 15707 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:06:38.398329 15707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:38.398344 15707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:38.398361 15707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:38.398386 15707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:38.398406 15707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:38.399823 15707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:38.400048 15707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:38.400250 15707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:06:38.404662 15707 server_base.cc:1034] running on GCE node
W20260924 05:06:38.404609 15715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.404631 15713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.404615 15712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:38.405133 15707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:38.405385 15707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:38.406513 15707 hybrid_clock.cc:648] HybridClock initialized: now 1790226398406500 us; error 32 us; skew 500 ppm
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:38.414991 15707 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:38.415432 15707 webserver.cc:460] Webserver started at http://127.0.98.62:34517/ using document root <none> and password file <none>
I20260924 05:06:38.415644 15707 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:38.415691 15707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:38.415817 15707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:38.416821 15707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "b2a58933eadeecec193578e2e4e446d0"
server_key_iv: "ff59192499135d089d162a417162d052"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.417150 15707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "b2a58933eadeecec193578e2e4e446d0"
server_key_iv: "ff59192499135d089d162a417162d052"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.418439 15707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:06:38.419085 15722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.419241 15707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:06:38.419324 15707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "b2a58933eadeecec193578e2e4e446d0"
server_key_iv: "ff59192499135d089d162a417162d052"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.419381 15707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:38.438687 15707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:38.440459 15707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:38.440593 15707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:38.444276 15707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46863
I20260924 05:06:38.444311 15774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46863 every 8 connection(s)
I20260924 05:06:38.444626 15707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:38.445405 15775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:38.447577 15775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap starting.
I20260924 05:06:38.448087 15775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:38.448312 15775 log.cc:824] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:38.448881 15775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: No bootstrap required, opened a new log
I20260924 05:06:38.450436 15775 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:38.450567 15775 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:38.450587 15775 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a76b8454a64cd0ad1795bfe7cdaa15, State: Initialized, Role: FOLLOWER
I20260924 05:06:38.450696 15775 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:38.450766 15775 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:38.450793 15775 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:38.450825 15775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:38.451251 15775 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:38.451340 15775 leader_election.cc:302] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9a76b8454a64cd0ad1795bfe7cdaa15; no voters:
I20260924 05:06:38.451457 15775 leader_election.cc:288] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:06:38.451500 15778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:38.451660 15775 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:38.451680 15778 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 LEADER]: Becoming Leader. State: Replica: e9a76b8454a64cd0ad1795bfe7cdaa15, State: Running, Role: LEADER
I20260924 05:06:38.451820 15778 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:38.452176 15780 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a76b8454a64cd0ad1795bfe7cdaa15. Latest consensus state: current_term: 1 leader_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260924 05:06:38.452308 15780 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:38.452183 15779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260924 05:06:38.452466 15779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:38.452754 15786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:38.453246 15786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:38.453415 15795 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:06:38.454083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15707
I20260924 05:06:38.454173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:06:38.454372 392 external_mini_cluster.cc:1428] Setting key 988fa319c0f4c6c6331f52c8cece6cfa
I20260924 05:06:38.454733 15786 catalog_manager.cc:1347] Generated new cluster ID: b81b5bc03ccb4d7f942ca05f82a8f5f6
I20260924 05:06:38.454778 15786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:38.472294 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.454955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.0.98.62:46863)
0924 05:06:38.455092 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.455101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.461944 (+ 6843us) server_negotiation.cc:407] Connection header received
0924 05:06:38.462068 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.462075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.462144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.462279 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:38.462984 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.464288 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.465347 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.465971 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.466055 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.468695 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:38.468699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:38.468699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:38.468711 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:38.470581 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.470938 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.470941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.470941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.471062 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.471221 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.471222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.471222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.471524 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:38.471667 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.471759 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.471890 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260924 05:06:38.483557 15786 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:06:38.484262 15786 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:38.490568 15786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Generated new TSK 0
I20260924 05:06:38.490754 15786 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:06:38.512364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:38.591039 15801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:38.591244 15801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:38.591275 15801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:38.592430 15801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:38.592469 15801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:38.592504 15801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:38.593940 15801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:38.594162 15801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:38.594368 15801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:38.599041 15809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.599041 15806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.599103 15807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:38.599285 15801 server_base.cc:1034] running on GCE node
I20260924 05:06:38.599476 15801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:38.599671 15801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:38.600822 15801 hybrid_clock.cc:648] HybridClock initialized: now 1790226398600808 us; error 30 us; skew 500 ppm
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:38.603158 15801 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:38.603549 15801 webserver.cc:460] Webserver started at http://127.0.98.1:38841/ using document root <none> and password file <none>
I20260924 05:06:38.603721 15801 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:38.603757 15801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:38.603845 15801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:38.604732 15801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "2060eb8d3ce649ef936d7ba70fc59d84"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "c2a65f0cf22dc06791eb44fd518274d2"
server_key_iv: "6b2e0d7c31e378e57c59a9900109ade6"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.605010 15801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2060eb8d3ce649ef936d7ba70fc59d84"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "c2a65f0cf22dc06791eb44fd518274d2"
server_key_iv: "6b2e0d7c31e378e57c59a9900109ade6"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.606132 15801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:06:38.606745 15816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.606909 15801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:38.606976 15801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "2060eb8d3ce649ef936d7ba70fc59d84"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "c2a65f0cf22dc06791eb44fd518274d2"
server_key_iv: "6b2e0d7c31e378e57c59a9900109ade6"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.607023 15801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:38.617755 15801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:38.619333 15801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:38.619437 15801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:38.619607 15801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:38.619833 15801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:38.619863 15801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.619886 15801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:38.619910 15801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.625129 15801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38427
I20260924 05:06:38.625195 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38427 every 8 connection(s)
I20260924 05:06:38.625465 15801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:38.626950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15801
I20260924 05:06:38.627063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:06:38.627244 392 external_mini_cluster.cc:1428] Setting key e88c7526d807ea4dbbc16ed77ba85ef8
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:38.635278 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.626174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49859 (local address 127.0.98.62:46863)
0924 05:06:38.626241 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.626243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.626431 (+ 188us) server_negotiation.cc:407] Connection header received
0924 05:06:38.626953 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.626955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.626967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.627020 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:38.628054 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.628558 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.629955 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.630160 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.630221 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.632617 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:38.632618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:38.632618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:38.632625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:38.633916 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.634368 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.634369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.634369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.634419 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.634673 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.634674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.634674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.634770 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:38.634811 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.635144 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.635210 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:06:38.635552 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.626144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:49859)
0924 05:06:38.626308 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:38.626348 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:38.626853 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:38.627093 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:38.627097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:38.627725 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:38.627963 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.627969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.628642 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.628643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.629874 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.629878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.630243 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.630244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.630397 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.630881 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:38.630892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:38.632524 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:38.633990 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:38.633996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:38.633998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:38.634291 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:38.634476 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:38.634477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:38.634477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:38.634614 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:38.634824 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:38.634831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:38.635014 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 05:06:38.635129 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 05:06:38.635296 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 05:06:38.636138 15930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260924 05:06:38.636222 15930 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:38.636386 15930 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260924 05:06:38.636889 15739 ts_manager.cc:194] Registered new tserver with Master: 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427)
I20260924 05:06:38.637497 15739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:06:38.649947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:38.728368 15935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:38.728583 15935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:38.728606 15935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:38.729751 15935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:38.729791 15935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:38.729822 15935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:38.731248 15935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:38.731448 15935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:38.731637 15935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:38.736349 15941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.736346 15940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.736343 15943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:38.736370 15935 server_base.cc:1034] running on GCE node
I20260924 05:06:38.736694 15935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:38.736927 15935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:38.738058 15935 hybrid_clock.cc:648] HybridClock initialized: now 1790226398738047 us; error 28 us; skew 500 ppm
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:38.740357 15935 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:38.740787 15935 webserver.cc:460] Webserver started at http://127.0.98.2:37737/ using document root <none> and password file <none>
I20260924 05:06:38.740994 15935 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:38.741039 15935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:38.741159 15935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:38.742203 15935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "d4e3d0f540d0a57c72ed04e5a8c8f71e"
server_key_iv: "4492a4b7127ae691de5f01eccb1d08fc"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.742518 15935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "d4e3d0f540d0a57c72ed04e5a8c8f71e"
server_key_iv: "4492a4b7127ae691de5f01eccb1d08fc"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.743673 15935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:06:38.744294 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.744431 15935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:38.744503 15935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "d4e3d0f540d0a57c72ed04e5a8c8f71e"
server_key_iv: "4492a4b7127ae691de5f01eccb1d08fc"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.744565 15935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:38.767896 15935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:38.769570 15935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:38.769694 15935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:38.769943 15935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:38.770229 15935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:38.770269 15935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.770290 15935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:38.770313 15935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.775564 15935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46377
I20260924 05:06:38.775619 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46377 every 8 connection(s)
I20260924 05:06:38.775898 15935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:38.783850 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.776534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37177 (local address 127.0.98.62:46863)
0924 05:06:38.776597 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.776599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.776777 (+ 178us) server_negotiation.cc:407] Connection header received
0924 05:06:38.777232 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.777233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.777242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.777303 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:38.778175 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.778666 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.779562 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.779739 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.779785 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.781628 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:38.781629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:38.781630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:38.781636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:38.782695 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.783041 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.783042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.783043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.783092 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.783291 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.783292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.783292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.783394 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:38.783448 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.783682 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.783762 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:06:38.784008 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.776498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:37177)
0924 05:06:38.776660 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:38.776702 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:38.777161 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:38.777363 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:38.777366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:38.777918 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:38.778104 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.778109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.778723 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.778723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.779504 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.779506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.779791 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.779792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.779882 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.780280 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:38.780287 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:38.781539 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:38.782771 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:38.782777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:38.782779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:38.782977 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:38.783144 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:38.783145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:38.783145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:38.783236 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:38.783447 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:38.783453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:38.783574 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 05:06:38.783657 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:06:38.783811 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260924 05:06:38.784286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 15935
I20260924 05:06:38.784379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:06:38.784438 16064 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260924 05:06:38.784523 16064 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:38.784571 392 external_mini_cluster.cc:1428] Setting key fec9fadf6afa8f5658c72ecf82e2dd34
I20260924 05:06:38.784686 16064 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260924 05:06:38.785004 15738 ts_manager.cc:194] Registered new tserver with Master: edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
I20260924 05:06:38.785351 15738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37177
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:06:38.805485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:38.883543 16069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:38.883728 16069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:38.883749 16069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:38.884837 16069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:38.884876 16069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:38.884907 16069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:38.886332 16069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:38.886546 16069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:38.886732 16069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:38.891127 16074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.891214 16075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:38.891124 16077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:38.891340 16069 server_base.cc:1034] running on GCE node
I20260924 05:06:38.891515 16069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:38.891696 16069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:38.892838 16069 hybrid_clock.cc:648] HybridClock initialized: now 1790226398892827 us; error 27 us; skew 500 ppm
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:38.895107 16069 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:38.895479 16069 webserver.cc:460] Webserver started at http://127.0.98.3:35233/ using document root <none> and password file <none>
I20260924 05:06:38.895666 16069 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:38.895707 16069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:38.895838 16069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:06:38.896760 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "52a86087108d4d65a010bc8ab5100296"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "311e68e110ed4b36ccd8a86a9f2809ce"
server_key_iv: "ff0357808cc34b5f91c2bab61e227338"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.897078 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "52a86087108d4d65a010bc8ab5100296"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "311e68e110ed4b36ccd8a86a9f2809ce"
server_key_iv: "ff0357808cc34b5f91c2bab61e227338"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.898240 16069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:06:38.898823 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.898972 16069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:38.899042 16069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "52a86087108d4d65a010bc8ab5100296"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "311e68e110ed4b36ccd8a86a9f2809ce"
server_key_iv: "ff0357808cc34b5f91c2bab61e227338"
server_key_version: "encryptionkey@0"
I20260924 05:06:38.899104 16069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:38.918319 16069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:38.920148 16069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:38.920274 16069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:38.920488 16069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:38.920771 16069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:38.920811 16069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.920836 16069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:38.920859 16069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:38.926313 16069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40899
I20260924 05:06:38.926348 16197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40899 every 8 connection(s)
I20260924 05:06:38.926651 16069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:06:38.929299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16069
I20260924 05:06:38.929406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:06:38.929594 392 external_mini_cluster.cc:1428] Setting key 1b3442cb3ac7611ce6f28240b50223e4
Sep 24 05:06:38 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:38.935068 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.927279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49341 (local address 127.0.98.62:46863)
0924 05:06:38.927354 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.927356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.927524 (+ 168us) server_negotiation.cc:407] Connection header received
0924 05:06:38.927986 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.927987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.927997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.928043 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:38.928936 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.929443 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.930764 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.930958 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.931006 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.932895 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:38.932897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:38.932897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:38.932903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:38.934023 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.934369 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.934370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.934370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.934418 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.934589 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.934589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.934590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.934685 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:38.934729 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.934941 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.935000 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:06:38.935242 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.927262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:49341)
0924 05:06:38.927413 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:38.927448 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:38.927911 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:38.928106 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:38.928109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:38.928661 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:38.928857 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.928862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.929523 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.929524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.930691 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.930694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.931013 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.931013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.931112 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.931539 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:38.931547 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:38.932808 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:38.934089 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:38.934095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:38.934097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:38.934313 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:38.934457 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:38.934458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:38.934458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:38.934551 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:38.934727 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:38.934730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:38.934842 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 05:06:38.934926 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:06:38.935044 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 05:06:38.935683 16198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260924 05:06:38.935771 16198 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:38.935925 16198 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260924 05:06:38.936259 15738 ts_manager.cc:194] Registered new tserver with Master: 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899)
I20260924 05:06:38.936622 15738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49341
I20260924 05:06:38.941208 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:06:38.947685 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.942466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.0.98.62:46863)
0924 05:06:38.942535 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.942537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.942606 (+ 69us) server_negotiation.cc:407] Connection header received
0924 05:06:38.942664 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.942665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.942674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.942720 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:38.943245 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.943711 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.944424 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.944611 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.944664 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.945690 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:38.945691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:38.945692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:38.945698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:38.946678 (+ 980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.946991 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.946992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.946992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.947045 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:38.947200 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:38.947201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:38.947201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:38.947345 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:38.947428 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.947527 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.947599 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:06:38.949080 15738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:06:38.950672 15738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:06:38.956588 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.952655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.0.1:40074)
0924 05:06:38.952970 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:38.952980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:38.953028 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:38.953310 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:38.953314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:38.953315 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:38.953374 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.953376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.954519 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.954520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.955409 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.955411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.956302 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.956303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.956382 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.956396 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:06:38.956466 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:06:38.956511 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
I20260924 05:06:38.956614 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.952801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.1:38427)
0924 05:06:38.953161 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.953166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.953175 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:06:38.953213 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.953216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.953231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.953298 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:38.953456 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.954434 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.955497 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.956220 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.956289 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.956347 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.956493 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.956532 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":80,"threads_started":1}
I20260924 05:06:38.957265 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.952653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40899 (local address 127.0.0.1:46614)
0924 05:06:38.952916 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:38.952958 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:38.953028 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:38.953854 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:38.953855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:38.953858 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:38.953911 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.953912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.955228 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.955229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.956065 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.956068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.957027 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.957028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.957102 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.957106 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:06:38.957158 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:06:38.957197 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260924 05:06:38.957285 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.952906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.0.98.3:40899)
0924 05:06:38.953609 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.953613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.953621 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:38.953654 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.953656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.953671 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.953839 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:38.953989 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.955117 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.956138 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.956824 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.957022 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.957084 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.957173 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.957208 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":77,"threads_started":1}
I20260924 05:06:38.957441 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.952898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46377 (local address 127.0.0.1:49096)
0924 05:06:38.953387 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:38.953396 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:38.953494 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:38.953864 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:38.953865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:38.953867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:38.953916 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.953918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.955205 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.955206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.956299 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:38.956302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.957213 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.957214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:38.957275 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.957280 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:06:38.957323 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:06:38.957361 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":25,"thread_start_us":50,"threads_started":1}
I20260924 05:06:38.957479 15864 tablet_service.cc:1467] Processing CreateTablet for tablet ca1df516fe77465a861b5d01a9a79db1 (DEFAULT_TABLE table=test-table [id=53ce1b36f15244f3a27bf8378de1fab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:38.957644 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:38.953073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.0.98.2:46377)
0924 05:06:38.953210 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:38.953214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:38.953457 (+ 243us) server_negotiation.cc:407] Connection header received
0924 05:06:38.953754 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:38.953756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:38.953772 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:38.953983 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:38.954055 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.955100 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.956389 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:38.957068 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:38.957129 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:38.957181 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:38.957534 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 05:06:38.957573 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260924 05:06:38.957861 15864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1df516fe77465a861b5d01a9a79db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:38.960299 16212 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap starting.
I20260924 05:06:38.960942 16212 tablet_bootstrap.cc:654] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:38.960893 16132 tablet_service.cc:1467] Processing CreateTablet for tablet ca1df516fe77465a861b5d01a9a79db1 (DEFAULT_TABLE table=test-table [id=53ce1b36f15244f3a27bf8378de1fab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:38.960976 15998 tablet_service.cc:1467] Processing CreateTablet for tablet ca1df516fe77465a861b5d01a9a79db1 (DEFAULT_TABLE table=test-table [id=53ce1b36f15244f3a27bf8378de1fab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:38.961156 16132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1df516fe77465a861b5d01a9a79db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:38.961318 15998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1df516fe77465a861b5d01a9a79db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:38.961535 16212 log.cc:824] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:38.962289 16212 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: No bootstrap required, opened a new log
I20260924 05:06:38.962361 16212 ts_tablet_manager.cc:1397] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 05:06:38.963598 16214 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Bootstrap starting.
I20260924 05:06:38.964177 16214 tablet_bootstrap.cc:654] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:38.964288 16212 raft_consensus.cc:348] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:38.964548 16215 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Bootstrap starting.
I20260924 05:06:38.964563 16212 raft_consensus.cc:374] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:38.964615 16212 raft_consensus.cc:729] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2060eb8d3ce649ef936d7ba70fc59d84, State: Initialized, Role: FOLLOWER
I20260924 05:06:38.964725 16214 log.cc:824] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:38.964766 16212 consensus_queue.cc:260] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:38.965042 15930 heartbeater.cc:503] Master 127.0.98.62:46863 was elected leader, sending a full tablet report...
I20260924 05:06:38.965160 16215 tablet_bootstrap.cc:654] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:38.965276 16212 ts_tablet_manager.cc:1428] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:06:38.965673 16215 log.cc:824] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:38.965850 16214 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: No bootstrap required, opened a new log
I20260924 05:06:38.965919 16214 ts_tablet_manager.cc:1397] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:06:38.966413 16215 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: No bootstrap required, opened a new log
I20260924 05:06:38.966480 16215 ts_tablet_manager.cc:1397] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:38.967805 16214 raft_consensus.cc:348] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:38.967974 16214 raft_consensus.cc:374] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:38.968047 16214 raft_consensus.cc:729] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a86087108d4d65a010bc8ab5100296, State: Initialized, Role: FOLLOWER
I20260924 05:06:38.968210 16214 consensus_queue.cc:260] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:38.968339 16215 raft_consensus.cc:348] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:38.968499 16215 raft_consensus.cc:374] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:38.968534 16215 raft_consensus.cc:729] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc92d3faeaf4534a6cfc7e198c630d1, State: Initialized, Role: FOLLOWER
I20260924 05:06:38.968580 16214 ts_tablet_manager.cc:1428] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:06:38.968597 16198 heartbeater.cc:503] Master 127.0.98.62:46863 was elected leader, sending a full tablet report...
I20260924 05:06:38.968657 16215 consensus_queue.cc:260] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:38.968915 16215 ts_tablet_manager.cc:1428] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:06:38.968976 16064 heartbeater.cc:503] Master 127.0.98.62:46863 was elected leader, sending a full tablet report...
W20260924 05:06:39.026417 16065 tablet.cc:2369] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:06:39.126145 15931 tablet.cc:2369] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:06:39.177186 16199 tablet.cc:2369] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:06:39.225983 16219 raft_consensus.cc:482] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:39.226125 16219 raft_consensus.cc:504] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:39.226485 16219 leader_election.cc:288] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377), 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427)
I20260924 05:06:39.229171 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.226755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50677 (local address 127.0.98.2:46377)
0924 05:06:39.226802 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:39.226804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:39.226814 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:06:39.226860 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:39.226861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:39.226879 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:39.226935 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:39.227115 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.227574 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.228486 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.228857 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.228916 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:39.228928 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:39.229068 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:06:39.229107 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:06:39.229171 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.226686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46377 (local address 127.0.98.3:50677)
0924 05:06:39.226731 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:39.226743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:39.226802 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:39.226966 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:39.226967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:39.226969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:39.227034 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:39.227036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.227654 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.227655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:39.228394 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:39.228396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.228931 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.228932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:39.229006 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:39.229010 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:06:39.229054 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:06:39.229095 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:06:39.229305 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.226741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37513 (local address 127.0.98.1:38427)
0924 05:06:39.226791 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:39.226793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:39.227074 (+ 281us) server_negotiation.cc:407] Connection header received
0924 05:06:39.227110 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:39.227111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:39.227124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:39.227171 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:39.227303 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.227752 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.228638 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.229003 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.229060 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:39.229070 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:39.229207 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 05:06:39.229244 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:06:39.229311 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.226686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.98.3:37513)
0924 05:06:39.227008 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:39.227017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:39.227066 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:39.227177 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:39.227177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:39.227179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:39.227246 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:39.227248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.227823 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.227824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:39.228557 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:39.228558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.229073 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.229074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:39.229147 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:39.229151 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:06:39.229194 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:06:39.229232 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":164,"thread_start_us":85,"threads_started":1}
I20260924 05:06:39.229571 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1df516fe77465a861b5d01a9a79db1" candidate_uuid: "52a86087108d4d65a010bc8ab5100296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" is_pre_election: true
I20260924 05:06:39.229660 15884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1df516fe77465a861b5d01a9a79db1" candidate_uuid: "52a86087108d4d65a010bc8ab5100296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" is_pre_election: true
I20260924 05:06:39.229753 16018 raft_consensus.cc:2454] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a86087108d4d65a010bc8ab5100296 in term 0.
I20260924 05:06:39.229828 15884 raft_consensus.cc:2454] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a86087108d4d65a010bc8ab5100296 in term 0.
I20260924 05:06:39.229965 16086 leader_election.cc:302] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1; no voters:
I20260924 05:06:39.230098 16219 raft_consensus.cc:2781] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:06:39.230165 16219 raft_consensus.cc:482] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:06:39.230180 16219 raft_consensus.cc:3037] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:39.230865 16219 raft_consensus.cc:504] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:39.230989 16219 leader_election.cc:288] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [CANDIDATE]: Term 1 election: Requested vote from peers edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377), 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427)
I20260924 05:06:39.231127 15884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1df516fe77465a861b5d01a9a79db1" candidate_uuid: "52a86087108d4d65a010bc8ab5100296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2060eb8d3ce649ef936d7ba70fc59d84"
I20260924 05:06:39.231128 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1df516fe77465a861b5d01a9a79db1" candidate_uuid: "52a86087108d4d65a010bc8ab5100296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
I20260924 05:06:39.231195 16018 raft_consensus.cc:3037] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:39.231199 15884 raft_consensus.cc:3037] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:39.231845 16018 raft_consensus.cc:2454] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a86087108d4d65a010bc8ab5100296 in term 1.
I20260924 05:06:39.231855 15884 raft_consensus.cc:2454] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a86087108d4d65a010bc8ab5100296 in term 1.
I20260924 05:06:39.232007 16088 leader_election.cc:302] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84, 52a86087108d4d65a010bc8ab5100296; no voters:
I20260924 05:06:39.232129 16219 raft_consensus.cc:2781] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:39.232276 16219 raft_consensus.cc:686] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 1 LEADER]: Becoming Leader. State: Replica: 52a86087108d4d65a010bc8ab5100296, State: Running, Role: LEADER
I20260924 05:06:39.232383 16219 consensus_queue.cc:237] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:39.233171 15738 catalog_manager.cc:5495] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a86087108d4d65a010bc8ab5100296 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "52a86087108d4d65a010bc8ab5100296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20260924 05:06:39.239435 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.237441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.0.98.3:40899)
0924 05:06:39.237488 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:39.237490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:39.237511 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:06:39.237537 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:39.237537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:39.237552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:39.237599 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:39.237706 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.238190 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.238856 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:39.239032 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:39.239073 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:39.239188 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:39.239282 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:06:39.239334 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:39.239350 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 05:06:39.239365 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:06:39.241053 15884 raft_consensus.cc:1261] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Refusing update from remote peer 52a86087108d4d65a010bc8ab5100296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:39.241087 16018 raft_consensus.cc:1261] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Refusing update from remote peer 52a86087108d4d65a010bc8ab5100296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:39.241386 16222 consensus_queue.cc:1035] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:39.241503 16219 consensus_queue.cc:1035] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:39.244451 16226 mvcc.cc:205] Tried to move back new op lower bound from 7332767331289006080 to 7332767331256426496. Current Snapshot: MvccSnapshot[applied={T|T < 7332767331289006080 or (T in {7332767331289006080})}]
I20260924 05:06:39.244653 16227 mvcc.cc:205] Tried to move back new op lower bound from 7332767331289006080 to 7332767331256426496. Current Snapshot: MvccSnapshot[applied={T|T < 7332767331289006080}]
I20260924 05:06:39.245610 15738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:39.245698 15738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:39.246407 15738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Sending DeleteTablet for 3 replicas of tablet ca1df516fe77465a861b5d01a9a79db1
I20260924 05:06:39.246675 16132 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1df516fe77465a861b5d01a9a79db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46614
I20260924 05:06:39.246737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 15801
I20260924 05:06:39.246757 15998 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1df516fe77465a861b5d01a9a79db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49096
I20260924 05:06:39.246856 15864 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1df516fe77465a861b5d01a9a79db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260924 05:06:39.247262 16234 tablet_replica.cc:330] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: stopping tablet replica
I20260924 05:06:39.247376 16234 raft_consensus.cc:2229] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:06:39.247483 16234 raft_consensus.cc:2258] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:39.247917 16234 ts_tablet_manager.cc:1905] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:39.249454 16234 ts_tablet_manager.cc:1918] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:39.249553 16234 log.cc:1196] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/ca1df516fe77465a861b5d01a9a79db1
I20260924 05:06:39.249857 16234 ts_tablet_manager.cc:1939] T ca1df516fe77465a861b5d01a9a79db1 P 52a86087108d4d65a010bc8ab5100296: Deleting consensus metadata
I20260924 05:06:39.250231 15724 catalog_manager.cc:4841] TS 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): tablet ca1df516fe77465a861b5d01a9a79db1 (table test-table [id=53ce1b36f15244f3a27bf8378de1fab5]) successfully deleted
W20260924 05:06:39.252574 15726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38427 (error 108)
I20260924 05:06:39.253010 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.252889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.0.1:40082)
0924 05:06:39.252940 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:39.252978 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:06:39.253012 16235 tablet_replica.cc:330] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: stopping tablet replica
I20260924 05:06:39.253103 16235 raft_consensus.cc:2229] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 05:06:39.253144 15726 catalog_manager.cc:4584] TS 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca1df516fe77465a861b5d01a9a79db1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38427: connect: Connection refused (error 111)
I20260924 05:06:39.253252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 15935
I20260924 05:06:39.257786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16069
I20260924 05:06:39.257820 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:39.257733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46377 (local address 127.0.0.1:49102)
0924 05:06:39.257772 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:39.257793 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 05:06:39.262048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 15707
I20260924 05:06:59.266769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--webserver_interface=127.0.98.62
--webserver_port=34517
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:06:59.344645 16237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:59.344848 16237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:59.344877 16237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:59.345986 16237 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:06:59.346021 16237 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:06:59.346038 16237 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:06:59.346053 16237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:06:59.346068 16237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:59.346086 16237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:59.346109 16237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:06:59.346133 16237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:06:59.347539 16237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:59.347769 16237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:59.347958 16237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:59.355338 16244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:59.355346 16243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.355495 16237 server_base.cc:1034] running on GCE node
W20260924 05:06:59.355346 16246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.355809 16237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:59.356076 16237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:59.357209 16237 hybrid_clock.cc:648] HybridClock initialized: now 1790226419357201 us; error 31 us; skew 500 ppm
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:59.363147 16237 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:06:59.363495 16237 webserver.cc:460] Webserver started at http://127.0.98.62:34517/ using document root <none> and password file <none>
I20260924 05:06:59.363672 16237 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:59.363719 16237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:59.364883 16237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:59.365413 16253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.365567 16237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:59.365646 16237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "b2a58933eadeecec193578e2e4e446d0"
server_key_iv: "ff59192499135d089d162a417162d052"
server_key_version: "encryptionkey@0"
I20260924 05:06:59.366017 16237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:59.382189 16237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:59.383754 16237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:59.383855 16237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:59.387420 16237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46863
I20260924 05:06:59.387452 16305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46863 every 8 connection(s)
I20260924 05:06:59.387806 16237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:06:59.389211 16306 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:06:59.389875 16306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap starting.
I20260924 05:06:59.391073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16237
I20260924 05:06:59.391328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38427
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46863
--webserver_port=38841
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:59.397773 16306 log.cc:824] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:59.399494 16306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:59.399731 16306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap complete.
I20260924 05:06:59.401757 16306 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:59.401994 16306 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a76b8454a64cd0ad1795bfe7cdaa15, State: Initialized, Role: FOLLOWER
I20260924 05:06:59.402154 16306 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:59.402222 16306 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:06:59.402273 16306 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:06:59.402318 16306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:06:59.403227 16306 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:59.403314 16306 leader_election.cc:302] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9a76b8454a64cd0ad1795bfe7cdaa15; no voters:
I20260924 05:06:59.403458 16306 leader_election.cc:288] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:06:59.403499 16310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:06:59.403656 16310 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 LEADER]: Becoming Leader. State: Replica: e9a76b8454a64cd0ad1795bfe7cdaa15, State: Running, Role: LEADER
I20260924 05:06:59.403710 16306 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:06:59.403820 16310 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:06:59.404136 16311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260924 05:06:59.404210 16311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:59.404546 16317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:06:59.404632 16312 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a76b8454a64cd0ad1795bfe7cdaa15. Latest consensus state: current_term: 2 leader_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260924 05:06:59.404772 16312 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:06:59.405344 16317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:06:59.405611 16317 catalog_manager.cc:1259] Loaded cluster ID: b81b5bc03ccb4d7f942ca05f82a8f5f6
I20260924 05:06:59.405685 16317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:06:59.406915 16317 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:06:59.407090 16317 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Loaded TSK: 0
I20260924 05:06:59.427938 16317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Generated new TSK 1
I20260924 05:06:59.428056 16317 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:06:59.476481 16308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:59.476661 16308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:59.476682 16308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:59.477813 16308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:59.477854 16308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:59.477877 16308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:06:59.479276 16308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38841
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:59.479472 16308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:59.479676 16308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:59.484108 16333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:59.484153 16336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:59.484207 16334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.484441 16308 server_base.cc:1034] running on GCE node
I20260924 05:06:59.484634 16308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:59.484838 16308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:59.485973 16308 hybrid_clock.cc:648] HybridClock initialized: now 1790226419485959 us; error 32 us; skew 500 ppm
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:59.488358 16308 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:06:59.488806 16308 webserver.cc:460] Webserver started at http://127.0.98.1:38841/ using document root <none> and password file <none>
I20260924 05:06:59.489027 16308 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:59.489086 16308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:59.490398 16308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:59.490988 16343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.491166 16308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:06:59.491261 16308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "2060eb8d3ce649ef936d7ba70fc59d84"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "c2a65f0cf22dc06791eb44fd518274d2"
server_key_iv: "6b2e0d7c31e378e57c59a9900109ade6"
server_key_version: "encryptionkey@0"
I20260924 05:06:59.491683 16308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:59.507289 16308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:59.509533 16308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:59.509693 16308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:59.509977 16308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:59.510458 16350 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:06:59.511847 16308 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:06:59.511907 16308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:06:59.511940 16308 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:06:59.512791 16308 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:06:59.512836 16308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:59.512883 16350 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap starting.
I20260924 05:06:59.519284 16308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38427
I20260924 05:06:59.519430 16457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38427 every 8 connection(s)
I20260924 05:06:59.519767 16308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:06:59.523443 16350 log.cc:824] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:59.525640 16350 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:59.525964 16350 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap complete.
I20260924 05:06:59.526125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16308
I20260924 05:06:59.526403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46377
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46863
--webserver_port=37737
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:59.527366 16350 ts_tablet_manager.cc:1397] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:59.528688 16350 raft_consensus.cc:348] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:59.529084 16350 raft_consensus.cc:729] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2060eb8d3ce649ef936d7ba70fc59d84, State: Initialized, Role: FOLLOWER
I20260924 05:06:59.529595 16350 consensus_queue.cc:260] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:59.530179 16350 ts_tablet_manager.cc:1428] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:06:59.532687 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.520563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35557 (local address 127.0.98.62:46863)
0924 05:06:59.521551 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.521560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.521604 (+ 44us) server_negotiation.cc:407] Connection header received
0924 05:06:59.521791 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.521801 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.521883 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.522052 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:59.522875 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.524081 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.525530 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.526175 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.526728 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.528876 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:59.528880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:59.528880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:59.528893 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:59.530722 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.531245 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.531247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.531248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.531356 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.531637 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.531638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.531638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.531910 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:59.531979 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.532345 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.532480 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":81,"threads_started":1}
I20260924 05:06:59.532691 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.520444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:35557)
0924 05:06:59.520622 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.520666 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.521323 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.522098 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.522102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.522436 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:59.522734 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.522742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.524198 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.524199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.525437 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.525440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.526291 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.526292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.526438 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.527086 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:59.527096 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:59.528763 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:59.530841 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.530848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.530851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.531151 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.531442 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.531443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.531443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.531576 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.531991 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:59.531998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:59.532168 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.532286 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.532482 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260924 05:06:59.533797 16458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260924 05:06:59.533885 16458 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:59.534055 16458 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260924 05:06:59.534734 16270 ts_manager.cc:194] Registered new tserver with Master: 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427)
I20260924 05:06:59.535732 16270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35557
I20260924 05:06:59.536646 16458 heartbeater.cc:503] Master 127.0.98.62:46863 was elected leader, sending a full tablet report...
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:06:59.543077 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.535368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.1:38427)
0924 05:06:59.535559 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.535563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.536105 (+ 542us) server_negotiation.cc:407] Connection header received
0924 05:06:59.536199 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.536202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.536219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.536286 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:59.536775 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.537457 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.538225 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.538514 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.538573 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.540570 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:59.540577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:59.540578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:59.540584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:59.541968 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.542345 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.542348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.542348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.542402 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.542562 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.542563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.542563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.542687 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:59.542751 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.542909 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.542987 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260924 05:06:59.543082 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.535290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.0.1:53438)
0924 05:06:59.535993 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.536033 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.536108 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.536298 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.536302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.536627 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:59.536687 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.536690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.537533 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.537534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.538156 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.538158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.538576 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.538579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.538655 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.539121 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:59.539125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:59.540494 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:59.542061 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.542068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.542070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.542255 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.542457 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.542458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.542458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.542513 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.542744 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:59.542750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:59.542798 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.542870 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.542979 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":92,"threads_started":1}
I20260924 05:06:59.543558 16392 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1df516fe77465a861b5d01a9a79db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53438
I20260924 05:06:59.544014 16467 tablet_replica.cc:330] stopping tablet replica
I20260924 05:06:59.544118 16467 raft_consensus.cc:2229] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:59.544183 16467 raft_consensus.cc:2258] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:59.544499 16467 ts_tablet_manager.cc:1905] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:59.546813 16467 ts_tablet_manager.cc:1918] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:59.546885 16467 log.cc:1196] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal/wals/ca1df516fe77465a861b5d01a9a79db1
I20260924 05:06:59.547086 16467 ts_tablet_manager.cc:1939] T ca1df516fe77465a861b5d01a9a79db1 P 2060eb8d3ce649ef936d7ba70fc59d84: Deleting consensus metadata
I20260924 05:06:59.547585 16257 catalog_manager.cc:4841] TS 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427): tablet ca1df516fe77465a861b5d01a9a79db1 (table test-table [id=53ce1b36f15244f3a27bf8378de1fab5]) successfully deleted
W20260924 05:06:59.608510 16463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:59.608688 16463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:59.608709 16463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:59.609843 16463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:59.609881 16463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:59.609910 16463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:06:59.611310 16463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37737
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:59.611515 16463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:59.611711 16463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:59.616101 16475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.616127 16463 server_base.cc:1034] running on GCE node
W20260924 05:06:59.616122 16473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:59.616111 16472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.616421 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:59.616603 16463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:59.617753 16463 hybrid_clock.cc:648] HybridClock initialized: now 1790226419617741 us; error 29 us; skew 500 ppm
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:59.620097 16463 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:06:59.620527 16463 webserver.cc:460] Webserver started at http://127.0.98.2:37737/ using document root <none> and password file <none>
I20260924 05:06:59.620716 16463 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:59.620766 16463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:59.622015 16463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:06:59.622573 16482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.622725 16463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:06:59.622802 16463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "d4e3d0f540d0a57c72ed04e5a8c8f71e"
server_key_iv: "4492a4b7127ae691de5f01eccb1d08fc"
server_key_version: "encryptionkey@0"
I20260924 05:06:59.623121 16463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:59.637295 16463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:59.639066 16463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:59.639174 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:59.639348 16463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:59.639739 16489 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:06:59.641103 16463 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:06:59.641142 16463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:06:59.641183 16463 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:06:59.641860 16463 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:06:59.641893 16463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 05:06:59.641975 16489 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Bootstrap starting.
I20260924 05:06:59.647071 16463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46377
I20260924 05:06:59.647169 16596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46377 every 8 connection(s)
I20260924 05:06:59.647424 16463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:06:59.651831 16489 log.cc:824] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:59.652103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16463
I20260924 05:06:59.652370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40899
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46863
--webserver_port=35233
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:06:59.654155 16489 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:06:59.654446 16489 tablet_bootstrap.cc:492] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Bootstrap complete.
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:59.655416 16489 ts_tablet_manager.cc:1397] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260924 05:06:59.656601 16489 raft_consensus.cc:348] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:59.656953 16489 raft_consensus.cc:729] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc92d3faeaf4534a6cfc7e198c630d1, State: Initialized, Role: FOLLOWER
I20260924 05:06:59.657117 16489 consensus_queue.cc:260] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } }
I20260924 05:06:59.657382 16489 ts_tablet_manager.cc:1428] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:06:59.657802 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.648165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59315 (local address 127.0.98.62:46863)
0924 05:06:59.648228 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.648230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.648467 (+ 237us) server_negotiation.cc:407] Connection header received
0924 05:06:59.649099 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.649100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.649113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.649153 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:59.650052 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.650527 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.652628 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.652842 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.652897 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.655185 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:59.655186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:59.655187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:59.655194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:59.656445 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.656787 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.656789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.656789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.656844 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.657103 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.657104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.657105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.657229 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:59.657284 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.657598 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.657686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:06:59.657909 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.648079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:59315)
0924 05:06:59.648308 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.648354 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.648992 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.649223 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.649228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.649634 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:59.649973 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.649980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.650629 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.650630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.652555 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.652558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.652925 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.652927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.653048 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.653479 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:59.653486 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:59.655102 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:59.656514 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.656521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.656523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.656723 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.656893 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.656894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.656894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.657016 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.657310 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:59.657317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:59.657460 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.657570 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.657733 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260924 05:06:59.658290 16597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260924 05:06:59.658396 16597 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:59.658563 16597 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260924 05:06:59.659067 16270 ts_manager.cc:194] Registered new tserver with Master: edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
I20260924 05:06:59.659549 16270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59315
I20260924 05:06:59.660320 16597 heartbeater.cc:503] Master 127.0.98.62:46863 was elected leader, sending a full tablet report...
I20260924 05:06:59.664697 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.659525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.0.98.2:46377)
0924 05:06:59.661798 (+ 2273us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.661801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.661809 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:59.661832 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.661834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.661845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.661902 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:59.662063 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.662648 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.664014 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.664416 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.664462 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.664494 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.664632 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.664655 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2247,"thread_start_us":77,"threads_started":1}
I20260924 05:06:59.664726 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.659324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46377 (local address 127.0.0.1:51646)
0924 05:06:59.659379 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.659392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.659442 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.661949 (+ 2507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.661950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.661953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:59.661997 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.661999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.662714 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.662715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.663941 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.663943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.664477 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.664478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.664548 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.664571 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.664614 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.664656 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:06:59.665089 16529 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1df516fe77465a861b5d01a9a79db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51646
I20260924 05:06:59.665448 16604 tablet_replica.cc:330] stopping tablet replica
I20260924 05:06:59.665544 16604 raft_consensus.cc:2229] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:59.665599 16604 raft_consensus.cc:2258] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:59.665843 16604 ts_tablet_manager.cc:1905] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:59.667706 16604 ts_tablet_manager.cc:1918] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:59.667763 16604 log.cc:1196] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/ca1df516fe77465a861b5d01a9a79db1
I20260924 05:06:59.667973 16604 ts_tablet_manager.cc:1939] T ca1df516fe77465a861b5d01a9a79db1 P edc92d3faeaf4534a6cfc7e198c630d1: Deleting consensus metadata
I20260924 05:06:59.668357 16255 catalog_manager.cc:4841] TS edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): tablet ca1df516fe77465a861b5d01a9a79db1 (table test-table [id=53ce1b36f15244f3a27bf8378de1fab5]) successfully deleted
W20260924 05:06:59.735857 16601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:06:59.736068 16601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:06:59.736090 16601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:06:59.737185 16601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:06:59.737218 16601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:06:59.737243 16601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:06:59.738692 16601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35233
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:06:59.738881 16601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:06:59.739058 16601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:06:59.743544 16609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.743724 16601 server_base.cc:1034] running on GCE node
W20260924 05:06:59.743551 16610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:06:59.743570 16612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:06:59.743978 16601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:06:59.744205 16601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:06:59.745335 16601 hybrid_clock.cc:648] HybridClock initialized: now 1790226419745320 us; error 26 us; skew 500 ppm
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:06:59.747471 16601 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:06:59.747829 16601 webserver.cc:460] Webserver started at http://127.0.98.3:35233/ using document root <none> and password file <none>
I20260924 05:06:59.748013 16601 fs_manager.cc:359] Metadata directory not provided
I20260924 05:06:59.748070 16601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:06:59.749254 16601 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:06:59.749706 16619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.749861 16601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.749936 16601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "52a86087108d4d65a010bc8ab5100296"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "311e68e110ed4b36ccd8a86a9f2809ce"
server_key_iv: "ff0357808cc34b5f91c2bab61e227338"
server_key_version: "encryptionkey@0"
I20260924 05:06:59.750238 16601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:06:59.770407 16601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:06:59.771939 16601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:06:59.772066 16601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:06:59.772253 16601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:06:59.772503 16601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:06:59.772536 16601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.772562 16601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:06:59.772579 16601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.777796 16601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40899
I20260924 05:06:59.777835 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40899 every 8 connection(s)
I20260924 05:06:59.778122 16601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:06:59 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:06:59.785689 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.778709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57301 (local address 127.0.98.62:46863)
0924 05:06:59.778786 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.778788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.778950 (+ 162us) server_negotiation.cc:407] Connection header received
0924 05:06:59.779382 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.779383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.779392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.779439 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:59.780236 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.780709 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.781579 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.781789 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.781839 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.783606 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:59.783607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:59.783607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:59.783613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:59.784774 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.785056 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.785056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.785057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.785098 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.785259 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.785259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.785260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.785343 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:59.785385 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.785576 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.785629 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:06:59.785867 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.778690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:57301)
0924 05:06:59.778833 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.778865 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.779309 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.779490 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.779493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.780001 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:06:59.780171 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.780175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.780767 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.780768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.781522 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.781524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.781847 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.781847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.781940 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.782301 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:06:59.782308 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:06:59.783529 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:06:59.784830 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.784832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.784834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.785008 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.785141 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:06:59.785141 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:06:59.785142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:06:59.785221 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:06:59.785382 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:06:59.785386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:06:59.785489 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.785558 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.785664 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260924 05:06:59.786231 16733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260924 05:06:59.786307 16733 heartbeater.cc:464] Registering TS with master...
I20260924 05:06:59.786445 16733 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260924 05:06:59.786752 16270 ts_manager.cc:194] Registered new tserver with Master: 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899)
I20260924 05:06:59.787087 16270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57301
I20260924 05:06:59.787441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16601
I20260924 05:06:59.792649 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.787788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.0.98.62:46863)
0924 05:06:59.787826 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.787828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.787971 (+ 143us) server_negotiation.cc:407] Connection header received
0924 05:06:59.788012 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.788013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.788020 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.788061 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:59.788547 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.788984 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.789653 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.789881 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.789924 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.790934 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:59.790935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:59.790935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:59.790939 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:59.791902 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.792168 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.792169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.792169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.792209 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.792319 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.792319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.792319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.792396 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:59.792513 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.792539 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.792591 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:06:59.793226 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:06:59.795542 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.793518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.0.98.62:46863)
0924 05:06:59.793571 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.793573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.793623 (+ 50us) server_negotiation.cc:407] Connection header received
0924 05:06:59.793688 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.793689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.793748 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.793807 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:59.793904 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.794351 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.795039 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.795221 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.795264 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.795361 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:59.795398 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 05:06:59.795491 (+ 93us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50072: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 05:06:59.795593 16461 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50072: token expired
W20260924 05:06:59.795682 16202 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46863)
I20260924 05:06:59.800143 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.795889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.0.98.62:46863)
0924 05:06:59.795929 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.795931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.795939 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:06:59.795960 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.795961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.795968 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.796026 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:06:59.796398 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.796843 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.797423 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.797582 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.797618 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.798500 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:06:59.798501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:06:59.798501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:06:59.798507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:06:59.799457 (+ 950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.799708 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.799709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.799709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.799750 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:06:59.799851 (+ 101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:06:59.799851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:06:59.799852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:06:59.799924 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:06:59.799981 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.800047 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.800093 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:06:59.801091 16270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:06:59.802649 16270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:06:59.804394 16392 tablet_service.cc:1467] Processing CreateTablet for tablet a5837fa754c6480aaa95115a91181523 (DEFAULT_TABLE table=test-table [id=d189808971334b7890b83134d996586a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:59.804589 16392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5837fa754c6480aaa95115a91181523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:59.804577 16529 tablet_service.cc:1467] Processing CreateTablet for tablet a5837fa754c6480aaa95115a91181523 (DEFAULT_TABLE table=test-table [id=d189808971334b7890b83134d996586a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:59.804737 16529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5837fa754c6480aaa95115a91181523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:59.805863 16489 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Bootstrap starting.
I20260924 05:06:59.806054 16350 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap starting.
I20260924 05:06:59.806159 16489 tablet_bootstrap.cc:654] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:59.806330 16350 tablet_bootstrap.cc:654] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:59.806707 16489 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: No bootstrap required, opened a new log
I20260924 05:06:59.806751 16489 ts_tablet_manager.cc:1397] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:59.806851 16489 raft_consensus.cc:348] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.806910 16489 raft_consensus.cc:374] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:59.806954 16489 raft_consensus.cc:729] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc92d3faeaf4534a6cfc7e198c630d1, State: Initialized, Role: FOLLOWER
I20260924 05:06:59.807004 16489 consensus_queue.cc:260] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.807093 16489 ts_tablet_manager.cc:1428] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:06:59.807399 16350 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: No bootstrap required, opened a new log
I20260924 05:06:59.807438 16350 ts_tablet_manager.cc:1397] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 05:06:59.807780 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.804146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40899 (local address 127.0.0.1:34314)
0924 05:06:59.804227 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.804237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.804266 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.804598 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.804599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.804601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:59.804633 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.804634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.805831 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.805832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.806731 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.806734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.807569 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.807569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.807624 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.807628 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.807680 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.807717 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 05:06:59.807796 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.804240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.98.3:40899)
0924 05:06:59.804412 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.804425 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.804434 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:06:59.804472 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.804475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.804488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.804593 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:59.804693 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.805739 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.806824 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.807505 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.807568 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.807624 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.807693 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.807725 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260924 05:06:59.807758 16350 raft_consensus.cc:348] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.807825 16350 raft_consensus.cc:374] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:59.807866 16350 raft_consensus.cc:729] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2060eb8d3ce649ef936d7ba70fc59d84, State: Initialized, Role: FOLLOWER
I20260924 05:06:59.807909 16350 consensus_queue.cc:260] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.807986 16350 ts_tablet_manager.cc:1428] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:06:59.808594 16667 tablet_service.cc:1467] Processing CreateTablet for tablet a5837fa754c6480aaa95115a91181523 (DEFAULT_TABLE table=test-table [id=d189808971334b7890b83134d996586a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:06:59.808883 16667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5837fa754c6480aaa95115a91181523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:06:59.811440 16741 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Bootstrap starting.
I20260924 05:06:59.811952 16741 tablet_bootstrap.cc:654] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Neither blocks nor log segments found. Creating new log.
I20260924 05:06:59.812386 16741 log.cc:824] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Log is configured to *not* fsync() on all Append() calls
I20260924 05:06:59.812985 16741 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: No bootstrap required, opened a new log
I20260924 05:06:59.813050 16741 ts_tablet_manager.cc:1397] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:06:59.814455 16741 raft_consensus.cc:348] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.814589 16741 raft_consensus.cc:374] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:06:59.814620 16741 raft_consensus.cc:729] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a86087108d4d65a010bc8ab5100296, State: Initialized, Role: FOLLOWER
I20260924 05:06:59.814723 16741 consensus_queue.cc:260] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.814934 16741 ts_tablet_manager.cc:1428] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:06:59.814980 16733 heartbeater.cc:503] Master 127.0.98.62:46863 was elected leader, sending a full tablet report...
I20260924 05:06:59.880216 16464 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:06:59.880340 16464 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.880625 16464 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
I20260924 05:06:59.883402 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.880866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35319 (local address 127.0.98.3:40899)
0924 05:06:59.880928 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.880930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.880939 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:06:59.880967 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.880968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.880979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.881041 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:59.881201 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.881759 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.882690 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.883079 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.883136 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.883146 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.883302 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.883334 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:06:59.883421 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.880949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51665 (local address 127.0.98.2:46377)
0924 05:06:59.881010 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.881013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.881132 (+ 119us) server_negotiation.cc:407] Connection header received
0924 05:06:59.881166 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.881167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.881179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.881223 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:06:59.881357 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.881864 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.882745 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.883083 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.883141 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.883150 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.883308 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.883344 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:06:59.883423 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.880774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40899 (local address 127.0.98.1:35319)
0924 05:06:59.880824 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.880835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.880886 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.881058 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.881059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.881061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:59.881136 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.881138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.881821 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.881822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.882611 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.882613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.883146 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.883147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.883225 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.883243 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.883290 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.883331 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:06:59.883440 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.880869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46377 (local address 127.0.98.1:51665)
0924 05:06:59.881066 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.881075 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:06:59.881125 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:06:59.881236 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:06:59.881236 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:06:59.881238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:06:59.881306 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.881308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.881928 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.881929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.882661 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:06:59.882663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.883162 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.883163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:06:59.883226 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.883244 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 05:06:59.883289 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:06:59.883328 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260924 05:06:59.883733 16551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5837fa754c6480aaa95115a91181523" candidate_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" is_pre_election: true
I20260924 05:06:59.883898 16551 raft_consensus.cc:2454] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2060eb8d3ce649ef936d7ba70fc59d84 in term 0.
I20260924 05:06:59.883937 16687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5837fa754c6480aaa95115a91181523" candidate_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a86087108d4d65a010bc8ab5100296" is_pre_election: true
I20260924 05:06:59.884070 16687 raft_consensus.cc:2454] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2060eb8d3ce649ef936d7ba70fc59d84 in term 0.
I20260924 05:06:59.884133 16345 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84, edc92d3faeaf4534a6cfc7e198c630d1; no voters:
I20260924 05:06:59.884266 16464 raft_consensus.cc:2781] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:06:59.884321 16464 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:06:59.884335 16464 raft_consensus.cc:3037] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:59.885079 16464 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.885190 16464 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 1 election: Requested vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
I20260924 05:06:59.885318 16687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5837fa754c6480aaa95115a91181523" candidate_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a86087108d4d65a010bc8ab5100296"
I20260924 05:06:59.885329 16551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5837fa754c6480aaa95115a91181523" candidate_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
I20260924 05:06:59.885385 16687 raft_consensus.cc:3037] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:59.885407 16551 raft_consensus.cc:3037] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:06:59.886083 16687 raft_consensus.cc:2454] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2060eb8d3ce649ef936d7ba70fc59d84 in term 1.
I20260924 05:06:59.886228 16345 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84, 52a86087108d4d65a010bc8ab5100296; no voters:
I20260924 05:06:59.886328 16464 raft_consensus.cc:2781] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:06:59.886476 16464 raft_consensus.cc:686] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 LEADER]: Becoming Leader. State: Replica: 2060eb8d3ce649ef936d7ba70fc59d84, State: Running, Role: LEADER
I20260924 05:06:59.886528 16551 raft_consensus.cc:2454] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2060eb8d3ce649ef936d7ba70fc59d84 in term 1.
I20260924 05:06:59.886550 16464 consensus_queue.cc:237] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:06:59.887279 16270 catalog_manager.cc:5495] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } health_report { overall_health: UNKNOWN } } }
W20260924 05:06:59.897915 16598 tablet.cc:2369] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:06:59.899811 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.897628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.1:38427)
0924 05:06:59.897677 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:06:59.897679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:06:59.897701 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:06:59.897754 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:06:59.897756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:06:59.897773 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:06:59.897834 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:06:59.897942 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.898443 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.899120 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:06:59.899342 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:06:59.899393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:06:59.899528 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:06:59.899623 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:06:59.899682 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:06:59.899723 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 05:06:59.899743 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:06:59.901124 16551 raft_consensus.cc:1261] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Refusing update from remote peer 2060eb8d3ce649ef936d7ba70fc59d84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:59.901137 16687 raft_consensus.cc:1261] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Refusing update from remote peer 2060eb8d3ce649ef936d7ba70fc59d84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:06:59.901412 16464 consensus_queue.cc:1035] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:59.901623 16745 consensus_queue.cc:1035] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:06:59.903342 16749 mvcc.cc:205] Tried to move back new op lower bound from 7332767415912665088 to 7332767415855828992. Current Snapshot: MvccSnapshot[applied={T|T < 7332767415912665088}]
I20260924 05:06:59.903781 16748 mvcc.cc:205] Tried to move back new op lower bound from 7332767415912665088 to 7332767415855828992. Current Snapshot: MvccSnapshot[applied={T|T < 7332767415912665088 or (T in {7332767415912665088})}]
I20260924 05:06:59.905422 16269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:59.905501 16269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:06:59.906175 16269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Sending DeleteTablet for 3 replicas of tablet a5837fa754c6480aaa95115a91181523
I20260924 05:06:59.906437 16529 tablet_service.cc:1514] Processing DeleteTablet for tablet a5837fa754c6480aaa95115a91181523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51646
I20260924 05:06:59.906450 16667 tablet_service.cc:1514] Processing DeleteTablet for tablet a5837fa754c6480aaa95115a91181523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:06:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34314
I20260924 05:06:59.906528 16604 tablet_replica.cc:330] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: stopping tablet replica
I20260924 05:06:59.906569 16604 raft_consensus.cc:2229] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:59.906616 16604 raft_consensus.cc:2258] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:59.906690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16308
I20260924 05:06:59.906634 16758 tablet_replica.cc:330] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: stopping tablet replica
I20260924 05:06:59.906880 16758 raft_consensus.cc:2229] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:06:59.906971 16758 raft_consensus.cc:2258] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:06:59.907298 16604 ts_tablet_manager.cc:1905] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:59.907437 16758 ts_tablet_manager.cc:1905] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:06:59.908406 16604 ts_tablet_manager.cc:1918] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:59.908439 16604 log.cc:1196] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/a5837fa754c6480aaa95115a91181523
I20260924 05:06:59.908583 16604 ts_tablet_manager.cc:1939] T a5837fa754c6480aaa95115a91181523 P edc92d3faeaf4534a6cfc7e198c630d1: Deleting consensus metadata
I20260924 05:06:59.908815 16255 catalog_manager.cc:4841] TS edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): tablet a5837fa754c6480aaa95115a91181523 (table test-table [id=d189808971334b7890b83134d996586a]) successfully deleted
I20260924 05:06:59.908854 16758 ts_tablet_manager.cc:1918] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:06:59.908911 16758 log.cc:1196] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/a5837fa754c6480aaa95115a91181523
I20260924 05:06:59.909152 16758 ts_tablet_manager.cc:1939] T a5837fa754c6480aaa95115a91181523 P 52a86087108d4d65a010bc8ab5100296: Deleting consensus metadata
I20260924 05:06:59.909476 16255 catalog_manager.cc:4841] TS 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): tablet a5837fa754c6480aaa95115a91181523 (table test-table [id=d189808971334b7890b83134d996586a]) successfully deleted
W20260924 05:06:59.912375 16257 connection.cc:535] client connection to 127.0.98.1:38427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 05:06:59.912467 16257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:06:59.912817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16463
I20260924 05:06:59.912881 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:06:59.912766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.0.1:53466)
0924 05:06:59.912815 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:06:59.912853 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:06:59.913564 16257 catalog_manager.cc:4584] TS 2060eb8d3ce649ef936d7ba70fc59d84 (127.0.98.1:38427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5837fa754c6480aaa95115a91181523: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38427: connect: Connection refused (error 111)
I20260924 05:06:59.916785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16601
I20260924 05:06:59.920853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16237
I20260924 05:07:19.925385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--webserver_interface=127.0.98.62
--webserver_port=34517
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:20.004024 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:20.004220 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:20.004247 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:20.005333 16760 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:07:20.005370 16760 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:07:20.005386 16760 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:07:20.005400 16760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:20.005416 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:20.005443 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:20.005466 16760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:20.005496 16760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:20.006944 16760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:20.007182 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:20.007380 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:20.012080 16768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.012116 16765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.012085 16766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:20.012245 16760 server_base.cc:1034] running on GCE node
I20260924 05:07:20.012463 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:20.012729 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:20.013857 16760 hybrid_clock.cc:648] HybridClock initialized: now 1790226440013844 us; error 29 us; skew 500 ppm
Sep 24 05:07:20 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:20.021303 16760 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:07:20.021813 16760 webserver.cc:460] Webserver started at http://127.0.98.62:34517/ using document root <none> and password file <none>
I20260924 05:07:20.022022 16760 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:20.022071 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:20.023357 16760 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:07:20.023950 16775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.024106 16760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:20.024187 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "b2a58933eadeecec193578e2e4e446d0"
server_key_iv: "ff59192499135d089d162a417162d052"
server_key_version: "encryptionkey@0"
I20260924 05:07:20.024554 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:20.033733 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:20.035847 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:20.035985 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:20.039820 16760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46863
I20260924 05:07:20.039853 16827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46863 every 8 connection(s)
I20260924 05:07:20.040208 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:20.041615 16828 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:07:20.042306 16828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap starting.
I20260924 05:07:20.050000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16760
I20260924 05:07:20.050223 16828 log.cc:824] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:20.050257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38427
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46863
--webserver_port=38841
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:07:20.052698 16828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:07:20.052989 16828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Bootstrap complete.
I20260924 05:07:20.056118 16828 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:07:20.056454 16828 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a76b8454a64cd0ad1795bfe7cdaa15, State: Initialized, Role: FOLLOWER
I20260924 05:07:20.056628 16828 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:07:20.056702 16828 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:20.056751 16828 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:20.056798 16828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:07:20.058172 16828 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:07:20.058283 16828 leader_election.cc:302] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9a76b8454a64cd0ad1795bfe7cdaa15; no voters:
I20260924 05:07:20.058455 16828 leader_election.cc:288] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 05:07:20.058501 16832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:07:20.058638 16832 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [term 3 LEADER]: Becoming Leader. State: Replica: e9a76b8454a64cd0ad1795bfe7cdaa15, State: Running, Role: LEADER
I20260924 05:07:20.058725 16828 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:20.058804 16832 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260924 05:07:20.059084 16833 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260924 05:07:20.059157 16833 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:20.059379 16834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a76b8454a64cd0ad1795bfe7cdaa15. Latest consensus state: current_term: 3 leader_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a76b8454a64cd0ad1795bfe7cdaa15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260924 05:07:20.059505 16834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:20.059628 16838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:20.060359 16838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:20.060451 16838 catalog_manager.cc:1259] Loaded cluster ID: b81b5bc03ccb4d7f942ca05f82a8f5f6
I20260924 05:07:20.060490 16838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:20.061409 16838 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:20.061579 16838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Loaded TSK: 0
I20260924 05:07:20.061615 16838 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e9a76b8454a64cd0ad1795bfe7cdaa15: Loaded TSK: 1
F20260924 05:07:20.061848 16838 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faee38a29ad google::LogMessage::Fail() at ??:0
@ 0x7faee38a588a google::LogMessage::SendToLog() at ??:0
@ 0x7faee38a24de google::LogMessage::Flush() at ??:0
@ 0x7faee38a3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faee600f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faee600cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faeeb7fc1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faeeb7fc48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faeeb7fc594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7faeeb7c2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7faeeb7ccad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faee57700d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faee5765954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faee65c96db start_thread at ??:0
@ 0x7faee23a061f clone at ??:0
W20260924 05:07:20.150449 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:20.150628 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:20.150648 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:20.151737 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:20.151777 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:20.151801 16831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:20.153174 16831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38841
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:20.153374 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:20.153571 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:20.157996 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:20.158165 16831 server_base.cc:1034] running on GCE node
W20260924 05:07:20.158006 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.158013 16872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:20.158483 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:20.158702 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:20.159834 16831 hybrid_clock.cc:648] HybridClock initialized: now 1790226440159820 us; error 31 us; skew 500 ppm
Sep 24 05:07:20 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:20.162031 16831 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:07:20.162424 16831 webserver.cc:460] Webserver started at http://127.0.98.1:38841/ using document root <none> and password file <none>
I20260924 05:07:20.162608 16831 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:20.162653 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:20.163874 16831 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:07:20.164502 16879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.164646 16831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.164721 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "2060eb8d3ce649ef936d7ba70fc59d84"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "c2a65f0cf22dc06791eb44fd518274d2"
server_key_iv: "6b2e0d7c31e378e57c59a9900109ade6"
server_key_version: "encryptionkey@0"
I20260924 05:07:20.165004 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:20.184027 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:20.186064 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:20.186195 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:20.186399 16831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:20.186830 16886 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:07:20.188742 16831 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:07:20.188784 16831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:07:20.188817 16831 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:07:20.189453 16831 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:07:20.189486 16831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:07:20.189579 16886 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap starting.
I20260924 05:07:20.195999 16831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38427
I20260924 05:07:20.196111 16993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38427 every 8 connection(s)
I20260924 05:07:20.196381 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:07:20.200136 16886 log.cc:824] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:20.202679 16886 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:07:20.202965 16886 tablet_bootstrap.cc:492] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Bootstrap complete.
I20260924 05:07:20.204011 16886 ts_tablet_manager.cc:1397] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260924 05:07:20.205338 16886 raft_consensus.cc:348] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:20.205677 16886 raft_consensus.cc:729] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2060eb8d3ce649ef936d7ba70fc59d84, State: Initialized, Role: FOLLOWER
I20260924 05:07:20.205767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16831
I20260924 05:07:20.205856 16886 consensus_queue.cc:260] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:20.206065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46377
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46863
--webserver_port=37737
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:07:20.206112 16886 ts_tablet_manager.cc:1428] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
W20260924 05:07:20.241372 16996 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:07:20.197001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:60751)
0924 05:07:20.197157 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.197198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:20.197658 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:20.241129 (+ 43471us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260924 05:07:20.241750 16994 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46863 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46863: Client connection negotiation failed: client connection to 127.0.98.62:46863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:07:20.242111 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.241992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:48309)
0924 05:07:20.242043 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.242086 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:07:20.242460 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.242371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:57791)
0924 05:07:20.242416 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.242441 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 05:07:20.242588 16994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:07:20.285290 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:20.285485 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:20.285506 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:20.286656 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:20.286703 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:20.286731 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:20.288177 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37737
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:20.288399 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:20.288612 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:20.293179 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.293156 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.293170 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:20.293359 16999 server_base.cc:1034] running on GCE node
I20260924 05:07:20.293541 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:20.293776 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:20.294900 16999 hybrid_clock.cc:648] HybridClock initialized: now 1790226440294886 us; error 26 us; skew 500 ppm
Sep 24 05:07:20 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:20.297040 16999 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:07:20.297408 16999 webserver.cc:460] Webserver started at http://127.0.98.2:37737/ using document root <none> and password file <none>
I20260924 05:07:20.297590 16999 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:20.297636 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:20.298853 16999 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:07:20.299324 17014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.299464 16999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.299540 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "edc92d3faeaf4534a6cfc7e198c630d1"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "d4e3d0f540d0a57c72ed04e5a8c8f71e"
server_key_iv: "4492a4b7127ae691de5f01eccb1d08fc"
server_key_version: "encryptionkey@0"
I20260924 05:07:20.299841 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:20.332718 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:20.334553 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:20.334681 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:20.334888 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:20.335186 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:20.335220 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.335248 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:20.335266 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.340569 16999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46377
I20260924 05:07:20.340629 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46377 every 8 connection(s)
I20260924 05:07:20.340924 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:20.341905 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.341511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:53861)
0924 05:07:20.341661 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.341766 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
W20260924 05:07:20.342262 17128 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46863 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46863: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
I20260924 05:07:20.342533 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.342452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:56329)
0924 05:07:20.342488 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.342515 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:07:20.342751 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.342691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:50649)
0924 05:07:20.342719 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.342738 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 05:07:20.342850 17128 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:07:20.350637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 16999
I20260924 05:07:20.350876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40899
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46863
--webserver_port=35233
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:20.427978 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:20.428202 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:20.428234 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:20.429397 17131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:20.429440 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:20.429476 17131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:20.430948 17131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35233
--tserver_master_addrs=127.0.98.62:46863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:20.431200 17131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:20.431413 17131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:20.436213 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.436239 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:20.436218 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:20.436365 17131 server_base.cc:1034] running on GCE node
I20260924 05:07:20.436584 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:20.436831 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:20.437968 17131 hybrid_clock.cc:648] HybridClock initialized: now 1790226440437952 us; error 34 us; skew 500 ppm
Sep 24 05:07:20 dist-test-slave-m01t krb5kdc[15694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:20.440204 17131 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:07:20.440634 17131 webserver.cc:460] Webserver started at http://127.0.98.3:35233/ using document root <none> and password file <none>
I20260924 05:07:20.440840 17131 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:20.440887 17131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:20.442222 17131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:07:20.442720 17146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.442855 17131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.442929 17131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "52a86087108d4d65a010bc8ab5100296"
format_stamp: "Formatted at 2026-09-24 05:06:38 on dist-test-slave-m01t"
server_key: "311e68e110ed4b36ccd8a86a9f2809ce"
server_key_iv: "ff0357808cc34b5f91c2bab61e227338"
server_key_version: "encryptionkey@0"
I20260924 05:07:20.443235 17131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:20.478962 17131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:20.480613 17131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:20.480718 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:20.480898 17131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:20.481134 17131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:20.481163 17131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.481186 17131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:20.481204 17131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:20.486670 17131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40899
I20260924 05:07:20.486720 17259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40899 every 8 connection(s)
I20260924 05:07:20.487042 17131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:07:20.488036 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.487671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:34723)
0924 05:07:20.487820 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.487890 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260924 05:07:20.488427 17260 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46863 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46863: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
I20260924 05:07:20.488665 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.488581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:40839)
0924 05:07:20.488621 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.488646 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:07:20.488876 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:20.488815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:46547)
0924 05:07:20.488843 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:20.488862 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 05:07:20.488956 17260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:07:20.495147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17131
I20260924 05:07:21.243105 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.242907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:34177)
0924 05:07:21.243009 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:21.243038 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260924 05:07:21.343319 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.343127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:47099)
0924 05:07:21.343233 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:21.343270 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260924 05:07:21.489459 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.489262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:36663)
0924 05:07:21.489376 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:21.489411 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260924 05:07:21.688361 17267 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:21.688519 17267 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:21.688818 17267 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
Sep 24 05:07:21 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 05:07:21 dist-test-slave-m01t krb5kdc[15694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 05:07:21.698604 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.689177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39903 (local address 127.0.98.2:46377)
0924 05:07:21.689309 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:21.689317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:21.689344 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:07:21.689891 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:21.689894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:21.689934 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:21.690073 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:21.690764 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.691608 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.692574 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.692957 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.693049 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:21.695094 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:21.695098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:21.695099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:21.695108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:21.696825 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697262 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:21.697265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:21.697265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:21.697365 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697603 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:21.697604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:21.697604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:21.697959 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:21.698037 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:21.698362 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 05:07:21.698490 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 05:07:21.698606 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.689068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56307 (local address 127.0.98.3:40899)
0924 05:07:21.689241 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:21.689254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:21.689293 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:07:21.689855 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:21.689858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:21.689904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:21.690019 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:21.690763 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.691646 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.692582 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.692956 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.693055 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:21.695414 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:21.695417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:21.695417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:21.695426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:21.696990 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697346 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:21.697348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:21.697348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:21.697439 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697602 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:21.697603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:21.697603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:21.697968 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:21.698037 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:21.698362 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 05:07:21.698490 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":51,"threads_started":1}
I20260924 05:07:21.698608 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.689077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46377 (local address 127.0.98.1:39903)
0924 05:07:21.689173 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:21.689182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:21.689219 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:21.690090 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:21.690093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:21.690466 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:21.690647 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:21.690654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.691698 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.691699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:21.692494 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:21.692496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.693034 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.693035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:21.693143 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:21.693626 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:21.693633 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:21.694994 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:21.696900 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:21.696906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:21.696907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:21.697185 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:21.697424 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:21.697425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:21.697541 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:21.698031 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:21.698039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:21.698193 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 05:07:21.698306 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 05:07:21.698497 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":6,"thread_start_us":52,"threads_started":1}
I20260924 05:07:21.698653 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:21.688977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40899 (local address 127.0.98.1:56307)
0924 05:07:21.689027 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:21.689038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:21.689094 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:21.690088 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:21.690092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:21.690465 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:21.690647 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:21.690654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.691737 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.691738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:21.692495 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:21.692497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:21.693037 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:21.693038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:21.693141 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:21.693683 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:21.693687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:21.695326 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:21.697056 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:21.697058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:21.697285 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:21.697494 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:21.697495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:21.697495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:21.697545 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:21.698032 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:21.698039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:21.698193 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 05:07:21.698306 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 05:07:21.698545 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":110}
W20260924 05:07:21.699430 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
W20260924 05:07:21.699522 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
I20260924 05:07:21.699555 16881 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84; no voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1
I20260924 05:07:21.699649 17267 raft_consensus.cc:2726] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:07:22.243697 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:22.243498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:50443)
0924 05:07:22.243620 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:22.243654 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260924 05:07:22.343911 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:22.343687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:52669)
0924 05:07:22.343831 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:22.343874 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":54,"threads_started":1}
I20260924 05:07:22.490039 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:22.489846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:53053)
0924 05:07:22.489962 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:22.489999 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 05:07:23.244352 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:23.244126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:38993)
0924 05:07:23.244242 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:23.244282 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260924 05:07:23.323305 17275 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:23.323405 17275 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:23.323535 17275 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
W20260924 05:07:23.323936 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
W20260924 05:07:23.324003 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
I20260924 05:07:23.324020 16881 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84; no voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1
I20260924 05:07:23.324088 17275 raft_consensus.cc:2726] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:07:23.344486 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:23.344306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:38879)
0924 05:07:23.344423 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:23.344450 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 05:07:23.490625 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:23.490415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:59899)
0924 05:07:23.490549 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:23.490585 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260924 05:07:24.245014 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:24.244782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:39735)
0924 05:07:24.244920 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:24.244966 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260924 05:07:24.345032 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:24.344841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:52917)
0924 05:07:24.344955 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:24.344992 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260924 05:07:24.491164 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:24.490992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:46041)
0924 05:07:24.491096 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:24.491127 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260924 05:07:24.948933 17281 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:24.949025 17281 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:24.949154 17281 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
W20260924 05:07:24.949501 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
W20260924 05:07:24.949565 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
I20260924 05:07:24.949580 16881 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84; no voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1
I20260924 05:07:24.949644 17281 raft_consensus.cc:2726] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:07:25.245615 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:25.245435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:46663)
0924 05:07:25.245537 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:25.245570 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260924 05:07:25.345547 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:25.345378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:49165)
0924 05:07:25.345479 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:25.345510 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260924 05:07:25.491703 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:25.491519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:46285)
0924 05:07:25.491627 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:25.491662 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260924 05:07:26.246160 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:26.245974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:34839)
0924 05:07:26.246083 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:26.246114 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 05:07:26.346091 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:26.345901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:48537)
0924 05:07:26.346017 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:26.346050 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260924 05:07:26.492237 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:26.492069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:41285)
0924 05:07:26.492173 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:26.492202 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260924 05:07:27.210505 17288 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:27.210595 17288 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:27.210714 17288 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
W20260924 05:07:27.211073 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
W20260924 05:07:27.211134 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
I20260924 05:07:27.211187 16881 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84; no voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1
I20260924 05:07:27.211251 17288 raft_consensus.cc:2726] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:07:27.246724 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:27.246526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:42047)
0924 05:07:27.246640 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:27.246679 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260924 05:07:27.346673 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:27.346477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:55197)
0924 05:07:27.346594 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:27.346630 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260924 05:07:27.492767 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:27.492584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:33815)
0924 05:07:27.492695 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:27.492727 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260924 05:07:28.247287 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:28.247098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:33867)
0924 05:07:28.247209 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:28.247241 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260924 05:07:28.347251 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:28.347057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:55657)
0924 05:07:28.347170 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:28.347206 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 05:07:28.493345 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:28.493163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:37231)
0924 05:07:28.493262 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:28.493298 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260924 05:07:29.247843 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:29.247647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:47591)
0924 05:07:29.247757 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:29.247797 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260924 05:07:29.347779 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:29.347584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:51911)
0924 05:07:29.347707 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:29.347741 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260924 05:07:29.493932 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:29.493747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:59491)
0924 05:07:29.493864 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:29.493895 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260924 05:07:30.248486 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:30.248280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:56175)
0924 05:07:30.248393 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:30.248434 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260924 05:07:30.348336 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:30.348147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:43821)
0924 05:07:30.348256 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:30.348297 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 05:07:30.494491 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:30.494309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:44971)
0924 05:07:30.494417 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:30.494449 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 05:07:31.249150 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:31.248940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:47183)
0924 05:07:31.249067 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:31.249104 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260924 05:07:31.348881 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:31.348692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:51701)
0924 05:07:31.348802 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:31.348837 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260924 05:07:31.439963 17303 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:31.440057 17303 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:31.440186 17303 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
W20260924 05:07:31.440551 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
W20260924 05:07:31.440622 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
I20260924 05:07:31.440637 16881 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84; no voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1
I20260924 05:07:31.440717 17303 raft_consensus.cc:2726] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:07:31.495038 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:31.494848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:55291)
0924 05:07:31.494959 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:31.494988 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260924 05:07:32.249784 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:32.249541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:50607)
0924 05:07:32.249669 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:32.249700 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260924 05:07:32.349483 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:32.349284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:58229)
0924 05:07:32.349394 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:32.349428 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260924 05:07:32.495633 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:32.495407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:57759)
0924 05:07:32.495553 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:32.495592 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
I20260924 05:07:33.250349 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:33.250132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:35431)
0924 05:07:33.250271 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:33.250305 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260924 05:07:33.350023 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:33.349854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:58313)
0924 05:07:33.349955 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:33.349985 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260924 05:07:33.496205 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:33.496001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:54427)
0924 05:07:33.496107 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:33.496142 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260924 05:07:34.125672 17311 raft_consensus.cc:482] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:34.125813 17311 raft_consensus.cc:504] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a86087108d4d65a010bc8ab5100296" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40899 } } peers { permanent_uuid: "2060eb8d3ce649ef936d7ba70fc59d84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "edc92d3faeaf4534a6cfc7e198c630d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46377 } }
I20260924 05:07:34.125976 17311 leader_election.cc:288] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899), edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377)
W20260924 05:07:34.126382 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a86087108d4d65a010bc8ab5100296 (127.0.98.3:40899): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
W20260924 05:07:34.126448 16881 leader_election.cc:341] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edc92d3faeaf4534a6cfc7e198c630d1 (127.0.98.2:46377): Not found: Tablet not found: a5837fa754c6480aaa95115a91181523
I20260924 05:07:34.126463 16881 leader_election.cc:302] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2060eb8d3ce649ef936d7ba70fc59d84; no voters: 52a86087108d4d65a010bc8ab5100296, edc92d3faeaf4534a6cfc7e198c630d1
I20260924 05:07:34.126541 17311 raft_consensus.cc:2726] T a5837fa754c6480aaa95115a91181523 P 2060eb8d3ce649ef936d7ba70fc59d84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:07:34.250916 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:34.250730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:35851)
0924 05:07:34.250839 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:34.250872 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260924 05:07:34.350556 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:34.350383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:48021)
0924 05:07:34.350483 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:34.350513 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260924 05:07:34.496706 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:34.496525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:59383)
0924 05:07:34.496626 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:34.496663 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260924 05:07:35.251515 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:35.251341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:44477)
0924 05:07:35.251441 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:35.251470 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260924 05:07:35.351128 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:35.350939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:56237)
0924 05:07:35.351055 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:35.351086 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260924 05:07:35.497222 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:35.497047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:33433)
0924 05:07:35.497158 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:35.497183 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260924 05:07:35.505452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16831
I20260924 05:07:35.509613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 16999
I20260924 05:07:35.514340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17131
2026-09-24T05:07:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58237 ms)
[----------] 30 tests from SecurityITest (226820 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:35 dist-test-slave-m01t krb5kdc[17320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:35 dist-test-slave-m01t krb5kdc[17320](info): set up 2 sockets
Sep 24 05:07:35 dist-test-slave-m01t krb5kdc[17320](info): commencing operation
krb5kdc: starting...
W20260924 05:07:36.457017 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:36 dist-test-slave-m01t krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:07:36.561522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:36.638870 17333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:36.639078 17333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:36.639099 17333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:36.640254 17333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:36.640295 17333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:36.640307 17333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:36.640321 17333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:36.640343 17333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:36.641811 17333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:36.642035 17333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:36.642239 17333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:36.646811 17339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:36.646821 17341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:36.646974 17333 server_base.cc:1034] running on GCE node
W20260924 05:07:36.646867 17338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:36.647267 17333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:36.647493 17333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:36.648624 17333 hybrid_clock.cc:648] HybridClock initialized: now 1790226456648613 us; error 29 us; skew 500 ppm
Sep 24 05:07:36 dist-test-slave-m01t krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:36.650835 17333 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:07:36.651225 17333 webserver.cc:460] Webserver started at http://127.0.98.62:32831/ using document root <none> and password file <none>
I20260924 05:07:36.651414 17333 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:36.651459 17333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:36.651589 17333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:36.652532 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "a648d427691f4533a4d7371b89706cb2"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "b91497361e59f3577ba8c847b53ae50b"
server_key_iv: "d8686c2481f770dab45b0355379c020b"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.652849 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "a648d427691f4533a4d7371b89706cb2"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "b91497361e59f3577ba8c847b53ae50b"
server_key_iv: "d8686c2481f770dab45b0355379c020b"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.654007 17333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:07:36.654594 17348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.654731 17333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:36.654806 17333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "a648d427691f4533a4d7371b89706cb2"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "b91497361e59f3577ba8c847b53ae50b"
server_key_iv: "d8686c2481f770dab45b0355379c020b"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.654862 17333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:36.673301 17333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:36.674808 17333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:36.674925 17333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:36.678480 17333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42913
I20260924 05:07:36.678509 17400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42913 every 8 connection(s)
I20260924 05:07:36.678797 17333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:36.679507 17401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:36.681501 17401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2: Bootstrap starting.
I20260924 05:07:36.682016 17401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:36.682269 17401 log.cc:824] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:36.682842 17401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2: No bootstrap required, opened a new log
I20260924 05:07:36.684309 17401 raft_consensus.cc:348] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648d427691f4533a4d7371b89706cb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260924 05:07:36.684432 17401 raft_consensus.cc:374] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:36.684461 17401 raft_consensus.cc:729] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648d427691f4533a4d7371b89706cb2, State: Initialized, Role: FOLLOWER
I20260924 05:07:36.684587 17401 consensus_queue.cc:260] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648d427691f4533a4d7371b89706cb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260924 05:07:36.684643 17401 raft_consensus.cc:388] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:36.684677 17401 raft_consensus.cc:482] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:36.684721 17401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:36.685163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17333
I20260924 05:07:36.685168 17401 raft_consensus.cc:504] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648d427691f4533a4d7371b89706cb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260924 05:07:36.685254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:36.685256 17401 leader_election.cc:302] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a648d427691f4533a4d7371b89706cb2; no voters:
I20260924 05:07:36.685400 17401 leader_election.cc:288] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:36.685431 392 external_mini_cluster.cc:1428] Setting key 933ebd1c3473d97d5182e26d9f10cf21
I20260924 05:07:36.685431 17404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:36.685534 17404 raft_consensus.cc:686] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [term 1 LEADER]: Becoming Leader. State: Replica: a648d427691f4533a4d7371b89706cb2, State: Running, Role: LEADER
I20260924 05:07:36.685637 17401 sys_catalog.cc:564] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:36.685664 17404 consensus_queue.cc:237] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648d427691f4533a4d7371b89706cb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } }
I20260924 05:07:36.686038 17404 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a648d427691f4533a4d7371b89706cb2. Latest consensus state: current_term: 1 leader_uuid: "a648d427691f4533a4d7371b89706cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648d427691f4533a4d7371b89706cb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } } }
I20260924 05:07:36.686179 17404 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:36.686192 17405 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a648d427691f4533a4d7371b89706cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648d427691f4533a4d7371b89706cb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42913 } } }
I20260924 05:07:36.686380 17405 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:36.686445 17415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:36.686982 17415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:36.688397 17415 catalog_manager.cc:1347] Generated new cluster ID: 5e5c0bd164c64fd5b2e260b1d6808676
I20260924 05:07:36.688444 17415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:07:36 dist-test-slave-m01t krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:36.696889 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:36.685995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.0.98.62:42913)
0924 05:07:36.686182 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:36.686191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:36.686221 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:07:36.686611 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:36.686619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:36.686679 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:36.686790 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:36.687944 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:36.689227 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:36.690316 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:36.690899 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:36.690993 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:36.693556 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:36.693560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:36.693561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:36.693573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:36.695422 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:36.695766 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:36.695769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:36.695770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:36.695882 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:36.696050 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:36.696050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:36.696051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:36.696331 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:36.696482 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:36.696556 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:07:36.696685 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260924 05:07:36.702107 17415 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:36.702786 17415 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:36.706146 17415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2: Generated new TSK 0
I20260924 05:07:36.706286 17415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:07:36.733413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:36.810281 17427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:36.810460 17427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:36.810488 17427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:36.811576 17427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:36.811614 17427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:36.811646 17427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:36.813019 17427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:36.813222 17427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:36.813405 17427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:36.817780 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:36.817780 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:36.817826 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:36.818059 17427 server_base.cc:1034] running on GCE node
I20260924 05:07:36.818280 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:36.818497 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:36.819618 17427 hybrid_clock.cc:648] HybridClock initialized: now 1790226456819610 us; error 26 us; skew 500 ppm
Sep 24 05:07:36 dist-test-slave-m01t krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:36.821998 17427 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:07:36.822417 17427 webserver.cc:460] Webserver started at http://127.0.98.1:42651/ using document root <none> and password file <none>
I20260924 05:07:36.822640 17427 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:36.822698 17427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:36.822835 17427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:36.823858 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "8efe3de199e241f6b5889e75f1856b1b"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "16fffd51893deff43d3a79008865f8e2"
server_key_iv: "7f54cc61b20ebdb756a764c383854989"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.824172 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8efe3de199e241f6b5889e75f1856b1b"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "16fffd51893deff43d3a79008865f8e2"
server_key_iv: "7f54cc61b20ebdb756a764c383854989"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.825342 17427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:36.826001 17442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.826139 17427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.826226 17427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "8efe3de199e241f6b5889e75f1856b1b"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "16fffd51893deff43d3a79008865f8e2"
server_key_iv: "7f54cc61b20ebdb756a764c383854989"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.826285 17427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:36.836238 17427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:36.838040 17427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:36.838166 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:36.838364 17427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:36.838645 17427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:36.838678 17427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.838707 17427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:36.838742 17427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.844072 17427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42189
I20260924 05:07:36.844130 17555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42189 every 8 connection(s)
I20260924 05:07:36.844403 17427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:07:36.847955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17427
I20260924 05:07:36.848049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:07:36.848237 392 external_mini_cluster.cc:1428] Setting key 3cd5d77ba317c5de1710532aa24fd2c8
Sep 24 05:07:36 dist-test-slave-m01t krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:36.853375 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:36.845003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50439 (local address 127.0.98.62:42913)
0924 05:07:36.845057 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:36.845059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:36.845230 (+ 171us) server_negotiation.cc:407] Connection header received
0924 05:07:36.846158 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:36.846160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:36.846176 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:36.846227 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:36.846822 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:36.847319 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:36.848342 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:36.848564 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:36.848635 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:36.850821 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:36.850822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:36.850822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:36.850830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:36.852182 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:36.852517 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:36.852518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:36.852519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:36.852575 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:36.852820 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:36.852820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:36.852821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:36.852914 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:36.852958 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:36.853256 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 05:07:36.853316 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:07:36.853644 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:36.844971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42913 (local address 127.0.98.1:50439)
0924 05:07:36.845105 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:36.845150 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:36.846048 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:36.846295 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:36.846298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:36.846540 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:36.846724 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:36.846729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:36.847399 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:36.847400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:36.848252 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:36.848255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:36.848643 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:36.848644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:36.848764 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:36.849221 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:36.849229 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:36.850740 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:36.852251 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:36.852257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:36.852258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:36.852453 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:36.852635 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:36.852636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:36.852636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:36.852761 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:36.852965 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:36.852970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:36.853130 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 05:07:36.853246 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 05:07:36.853406 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260924 05:07:36.854139 17556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42913
I20260924 05:07:36.854233 17556 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:36.854391 17556 heartbeater.cc:511] Master 127.0.98.62:42913 requested a full tablet report, sending...
I20260924 05:07:36.854856 17365 ts_manager.cc:194] Registered new tserver with Master: 8efe3de199e241f6b5889e75f1856b1b (127.0.98.1:42189)
I20260924 05:07:36.855403 17365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50439
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:07:36.870147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:36.948362 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:36.948551 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:36.948572 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:36.949676 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:36.949739 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:36.949781 17561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:36.951207 17561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:36.951399 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:36.951583 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:36.955976 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:36.956000 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:36.956032 17561 server_base.cc:1034] running on GCE node
W20260924 05:07:36.955996 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:36.956377 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:36.956583 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:36.957742 17561 hybrid_clock.cc:648] HybridClock initialized: now 1790226456957701 us; error 26 us; skew 500 ppm
Sep 24 05:07:36 dist-test-slave-m01t krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:36.959941 17561 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:07:36.960312 17561 webserver.cc:460] Webserver started at http://127.0.98.2:43057/ using document root <none> and password file <none>
I20260924 05:07:36.960486 17561 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:36.960528 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:36.960640 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:36.961545 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "418df36e55f740cb86162d1a96ac0926"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "b582ca90531321c087824adf3164ebe6"
server_key_iv: "2743c4c8af15f3d8d3476ac38d1d9706"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.961869 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "418df36e55f740cb86162d1a96ac0926"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "b582ca90531321c087824adf3164ebe6"
server_key_iv: "2743c4c8af15f3d8d3476ac38d1d9706"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.962970 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:07:36.963569 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.963706 17561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:36.963786 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "418df36e55f740cb86162d1a96ac0926"
format_stamp: "Formatted at 2026-09-24 05:07:36 on dist-test-slave-m01t"
server_key: "b582ca90531321c087824adf3164ebe6"
server_key_iv: "2743c4c8af15f3d8d3476ac38d1d9706"
server_key_version: "encryptionkey@0"
I20260924 05:07:36.963842 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:36.991365 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:36.992950 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:36.993075 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:36.993269 17561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:36.993549 17561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:36.993582 17561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.993608 17561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:36.993625 17561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:36.998857 17561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36601
I20260924 05:07:36.998915 17689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36601 every 8 connection(s)
I20260924 05:07:36.999171 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:37.004014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17561
I20260924 05:07:37.004134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:07:37.004338 392 external_mini_cluster.cc:1428] Setting key 9fa8e0ba79390beaada860f51b4ec1cc
Sep 24 05:07:37 dist-test-slave-m01t krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:37.008618 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:36.999734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57371 (local address 127.0.98.62:42913)
0924 05:07:36.999812 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:36.999814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:36.999928 (+ 114us) server_negotiation.cc:407] Connection header received
0924 05:07:37.000785 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.000786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.000795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.000840 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:37.001358 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.001852 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.002737 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.002906 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.002946 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.006068 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:37.006069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:37.006070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:37.006075 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:37.007439 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:37.007773 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:37.007774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:37.007775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:37.007831 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:37.008049 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:37.008050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:37.008050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:37.008180 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:37.008246 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.008445 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.008525 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:07:37.008666 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:36.999699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42913 (local address 127.0.98.2:57371)
0924 05:07:36.999835 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:36.999867 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.000702 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.000892 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.000895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.001118 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:37.001299 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.001303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.001904 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.001904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.002677 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.002679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.002955 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.002955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.003043 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.003467 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:37.003474 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:37.004909 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:37.007507 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:37.007513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:37.007514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:37.007710 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:37.007888 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:37.007888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:37.007889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:37.007985 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:37.008235 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:37.008238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:37.008340 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.008412 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.008518 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260924 05:07:37.009093 17690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42913
I20260924 05:07:37.009181 17690 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:37.009334 17690 heartbeater.cc:511] Master 127.0.98.62:42913 requested a full tablet report, sending...
I20260924 05:07:37.009654 17365 ts_manager.cc:194] Registered new tserver with Master: 418df36e55f740cb86162d1a96ac0926 (127.0.98.2:36601)
I20260924 05:07:37.010010 17365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57371
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:07:37.025696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:37.103049 17695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:37.103236 17695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:37.103266 17695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:37.104342 17695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:37.104380 17695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:37.104413 17695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:37.105811 17695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:37.106014 17695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:37.106205 17695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:07:37.110697 17695 server_base.cc:1034] running on GCE node
W20260924 05:07:37.110677 17701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.110672 17703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.110663 17700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:37.111071 17695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:37.111297 17695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:37.112429 17695 hybrid_clock.cc:648] HybridClock initialized: now 1790226457112414 us; error 26 us; skew 500 ppm
Sep 24 05:07:37 dist-test-slave-m01t krb5kdc[17320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:37.114697 17695 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:07:37.115056 17695 webserver.cc:460] Webserver started at http://127.0.98.3:38603/ using document root <none> and password file <none>
I20260924 05:07:37.115252 17695 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:37.115298 17695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:37.115412 17695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:37.116353 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe48104090ac4a3ca26cb7ed828afbca"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "f0a7ff03447ffd7727a172f7d5b26438"
server_key_iv: "30725ce2f02a95a12aef3b92fa3ced33"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.116659 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe48104090ac4a3ca26cb7ed828afbca"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "f0a7ff03447ffd7727a172f7d5b26438"
server_key_iv: "30725ce2f02a95a12aef3b92fa3ced33"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.117823 17695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:37.118439 17710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.118582 17695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.118657 17695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "fe48104090ac4a3ca26cb7ed828afbca"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "f0a7ff03447ffd7727a172f7d5b26438"
server_key_iv: "30725ce2f02a95a12aef3b92fa3ced33"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.118721 17695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:37.149649 17695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:37.151834 17695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:37.151974 17695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:37.152201 17695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:37.152488 17695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:37.152521 17695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.152550 17695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:37.152571 17695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.158073 17695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34635
I20260924 05:07:37.158115 17823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34635 every 8 connection(s)
I20260924 05:07:37.158442 17695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:07:37.159940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17695
I20260924 05:07:37.160029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:07:37.160202 392 external_mini_cluster.cc:1428] Setting key da8dd5296e55d75d0d8b58ddff984e12
Sep 24 05:07:37 dist-test-slave-m01t krb5kdc[17320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:37.166538 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.159064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48879 (local address 127.0.98.62:42913)
0924 05:07:37.159121 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.159123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.159322 (+ 199us) server_negotiation.cc:407] Connection header received
0924 05:07:37.160385 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.160386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.160398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.160445 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:37.161037 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.161518 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.162464 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.162653 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.162699 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.164459 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:37.164460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:37.164460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:37.164467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:37.165490 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:37.165839 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:37.165839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:37.165840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:37.165885 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:37.166049 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:37.166050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:37.166050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:37.166158 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:37.166202 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.166408 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.166463 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:07:37.166713 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.159067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42913 (local address 127.0.98.3:48879)
0924 05:07:37.159203 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.159248 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.160273 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.160519 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.160523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.160793 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:37.160976 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.160980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.161578 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.161579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.162405 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.162407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.162707 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.162707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.162799 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.163181 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:37.163187 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:37.164384 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:37.165549 (+ 1165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:37.165554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:37.165556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:37.165792 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:37.165929 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:37.165929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:37.165930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:37.166013 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:37.166201 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:37.166204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:37.166312 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.166393 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.166498 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260924 05:07:37.167083 17824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42913
I20260924 05:07:37.167160 17824 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:37.167330 17824 heartbeater.cc:511] Master 127.0.98.62:42913 requested a full tablet report, sending...
I20260924 05:07:37.167655 17365 ts_manager.cc:194] Registered new tserver with Master: fe48104090ac4a3ca26cb7ed828afbca (127.0.98.3:34635)
I20260924 05:07:37.167990 17365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48879
I20260924 05:07:37.171569 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:07:37.177417 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.172772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.98.62:42913)
0924 05:07:37.172811 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.172813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.172908 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:07:37.172944 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.172945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.172952 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.172992 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:37.173445 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.173927 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.174547 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.174717 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.174754 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.175685 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:37.175686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:37.175686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:37.175690 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:37.176656 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:37.176910 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:37.176911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:37.176911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:37.176949 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:37.177061 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:37.177061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:37.177061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:37.177174 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:37.177239 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.177312 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.177360 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:07:37.178695 17365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:07:37.180133 17365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:07:37.185647 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.182060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42189 (local address 127.0.0.1:40770)
0924 05:07:37.182276 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.182285 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.182322 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.182513 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.182514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.182517 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:37.182561 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.182563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.183801 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.183802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.184636 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.184639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.185413 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.185414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.185473 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.185483 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.185527 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.185560 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":52,"thread_start_us":44,"threads_started":1}
I20260924 05:07:37.185699 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.182201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.98.1:42189)
0924 05:07:37.182375 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.182379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.182387 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:37.182420 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.182422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.182436 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.182513 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:37.182615 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.183707 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.184722 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.185338 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.185403 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.185456 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.185541 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.185578 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20260924 05:07:37.186149 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.182174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34635 (local address 127.0.0.1:35108)
0924 05:07:37.182529 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.182538 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.182606 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.182876 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.182877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.182879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:37.182917 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.182919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.183708 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.183709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.184989 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.184990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.185946 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.185947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.186008 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.186013 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.186065 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.186096 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":69,"thread_start_us":67,"threads_started":1}
I20260924 05:07:37.186201 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.182335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.98.3:34635)
0924 05:07:37.182483 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.182487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.182584 (+ 97us) server_negotiation.cc:407] Connection header received
0924 05:07:37.182667 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.182669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.182681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.182806 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:37.182986 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.183619 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.185060 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.185756 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.185818 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.185874 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.186091 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.186130 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260924 05:07:37.186367 17490 tablet_service.cc:1467] Processing CreateTablet for tablet 0c474ddbecba4268ac8371bc9574f082 (DEFAULT_TABLE table=test-table [id=d05973cce70842c3b8007accdb0b126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:37.186749 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c474ddbecba4268ac8371bc9574f082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:37.186774 17758 tablet_service.cc:1467] Processing CreateTablet for tablet 0c474ddbecba4268ac8371bc9574f082 (DEFAULT_TABLE table=test-table [id=d05973cce70842c3b8007accdb0b126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:37.187058 17758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c474ddbecba4268ac8371bc9574f082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:37.187438 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.181889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36601 (local address 127.0.0.1:59752)
0924 05:07:37.181996 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.182037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.182102 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.182404 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.182406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.182408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:37.182465 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.182466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.183747 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.183748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.184981 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.184984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.187105 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.187106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.187177 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.187181 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.187359 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.187393 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":43,"thread_start_us":53,"threads_started":1}
I20260924 05:07:37.187757 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.181996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.98.2:36601)
0924 05:07:37.182210 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.182227 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.182236 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:37.182275 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.182277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.182291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.182398 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:37.182543 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.183660 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.186292 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.187024 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.187580 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.187646 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.187670 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.187700 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260924 05:07:37.188251 17624 tablet_service.cc:1467] Processing CreateTablet for tablet 0c474ddbecba4268ac8371bc9574f082 (DEFAULT_TABLE table=test-table [id=d05973cce70842c3b8007accdb0b126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:37.188508 17624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c474ddbecba4268ac8371bc9574f082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:37.189968 17839 tablet_bootstrap.cc:492] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: Bootstrap starting.
I20260924 05:07:37.190181 17838 tablet_bootstrap.cc:492] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Bootstrap starting.
I20260924 05:07:37.190807 17838 tablet_bootstrap.cc:654] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:37.191107 17840 tablet_bootstrap.cc:492] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Bootstrap starting.
I20260924 05:07:37.191390 17838 log.cc:824] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:37.191632 17840 tablet_bootstrap.cc:654] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:37.192080 17840 log.cc:824] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:37.192113 17838 tablet_bootstrap.cc:492] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: No bootstrap required, opened a new log
I20260924 05:07:37.192135 17839 tablet_bootstrap.cc:654] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:37.192171 17838 ts_tablet_manager.cc:1397] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:37.192625 17840 tablet_bootstrap.cc:492] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: No bootstrap required, opened a new log
I20260924 05:07:37.192610 17839 log.cc:824] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:37.192680 17840 ts_tablet_manager.cc:1397] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:37.193312 17839 tablet_bootstrap.cc:492] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: No bootstrap required, opened a new log
I20260924 05:07:37.193382 17839 ts_tablet_manager.cc:1397] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:07:37.194104 17838 raft_consensus.cc:348] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.194147 17840 raft_consensus.cc:348] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.194270 17840 raft_consensus.cc:374] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:37.194290 17840 raft_consensus.cc:729] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418df36e55f740cb86162d1a96ac0926, State: Initialized, Role: FOLLOWER
I20260924 05:07:37.194315 17838 raft_consensus.cc:374] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:37.194378 17838 raft_consensus.cc:729] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe48104090ac4a3ca26cb7ed828afbca, State: Initialized, Role: FOLLOWER
I20260924 05:07:37.194434 17840 consensus_queue.cc:260] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.194737 17690 heartbeater.cc:503] Master 127.0.98.62:42913 was elected leader, sending a full tablet report...
I20260924 05:07:37.194514 17838 consensus_queue.cc:260] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.194739 17840 ts_tablet_manager.cc:1428] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:37.195020 17838 ts_tablet_manager.cc:1428] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:07:37.195171 17824 heartbeater.cc:503] Master 127.0.98.62:42913 was elected leader, sending a full tablet report...
I20260924 05:07:37.195326 17839 raft_consensus.cc:348] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.195482 17839 raft_consensus.cc:374] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:37.195573 17839 raft_consensus.cc:729] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efe3de199e241f6b5889e75f1856b1b, State: Initialized, Role: FOLLOWER
I20260924 05:07:37.195713 17839 consensus_queue.cc:260] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.195981 17839 ts_tablet_manager.cc:1428] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:07:37.196014 17556 heartbeater.cc:503] Master 127.0.98.62:42913 was elected leader, sending a full tablet report...
W20260924 05:07:37.249662 17691 tablet.cc:2369] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:07:37.345044 17557 tablet.cc:2369] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:07:37.408986 17825 tablet.cc:2369] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:07:37.474128 17844 raft_consensus.cc:482] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:37.474279 17844 raft_consensus.cc:504] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.474610 17844 leader_election.cc:288] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8efe3de199e241f6b5889e75f1856b1b (127.0.98.1:42189), fe48104090ac4a3ca26cb7ed828afbca (127.0.98.3:34635)
I20260924 05:07:37.477166 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.474831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33727 (local address 127.0.98.1:42189)
0924 05:07:37.474889 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.474891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.474899 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:37.474927 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.474927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.474937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.474993 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:37.475137 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.475604 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.476493 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.476843 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.476905 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.476915 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.477073 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.477108 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:07:37.477165 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.474755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42189 (local address 127.0.98.2:33727)
0924 05:07:37.474801 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.474813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.474853 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.475008 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.475009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.475012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:37.475077 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.475079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.475675 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.475675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.476407 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.476408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.476919 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.476920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.477011 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.477014 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.477059 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.477094 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":1}
I20260924 05:07:37.477231 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.474950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47373 (local address 127.0.98.3:34635)
0924 05:07:37.475012 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.475014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.475021 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:07:37.475047 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.475048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.475054 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.475096 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:37.475247 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.475700 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.476565 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.476925 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.476980 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.476990 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.477132 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.477165 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:07:37.477280 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.474856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34635 (local address 127.0.98.2:47373)
0924 05:07:37.474933 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.474941 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.474973 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.475109 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.475110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.475111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:37.475186 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.475187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.475771 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.475772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.476491 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.476492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.477019 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.477020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.477075 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.477078 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.477128 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.477164 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":52,"threads_started":1}
I20260924 05:07:37.477488 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c474ddbecba4268ac8371bc9574f082" candidate_uuid: "418df36e55f740cb86162d1a96ac0926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efe3de199e241f6b5889e75f1856b1b" is_pre_election: true
I20260924 05:07:37.477658 17510 raft_consensus.cc:2454] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418df36e55f740cb86162d1a96ac0926 in term 0.
I20260924 05:07:37.477631 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c474ddbecba4268ac8371bc9574f082" candidate_uuid: "418df36e55f740cb86162d1a96ac0926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48104090ac4a3ca26cb7ed828afbca" is_pre_election: true
I20260924 05:07:37.477789 17778 raft_consensus.cc:2454] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418df36e55f740cb86162d1a96ac0926 in term 0.
I20260924 05:07:37.477905 17577 leader_election.cc:302] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418df36e55f740cb86162d1a96ac0926, 8efe3de199e241f6b5889e75f1856b1b; no voters:
I20260924 05:07:37.478014 17844 raft_consensus.cc:2781] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:07:37.478084 17844 raft_consensus.cc:482] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:07:37.478101 17844 raft_consensus.cc:3037] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:37.478749 17844 raft_consensus.cc:504] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.478850 17844 leader_election.cc:288] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [CANDIDATE]: Term 1 election: Requested vote from peers 8efe3de199e241f6b5889e75f1856b1b (127.0.98.1:42189), fe48104090ac4a3ca26cb7ed828afbca (127.0.98.3:34635)
I20260924 05:07:37.479012 17778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c474ddbecba4268ac8371bc9574f082" candidate_uuid: "418df36e55f740cb86162d1a96ac0926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48104090ac4a3ca26cb7ed828afbca"
I20260924 05:07:37.479002 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c474ddbecba4268ac8371bc9574f082" candidate_uuid: "418df36e55f740cb86162d1a96ac0926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efe3de199e241f6b5889e75f1856b1b"
I20260924 05:07:37.479084 17778 raft_consensus.cc:3037] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:37.479085 17510 raft_consensus.cc:3037] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:37.479887 17510 raft_consensus.cc:2454] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418df36e55f740cb86162d1a96ac0926 in term 1.
I20260924 05:07:37.479887 17778 raft_consensus.cc:2454] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418df36e55f740cb86162d1a96ac0926 in term 1.
I20260924 05:07:37.480024 17577 leader_election.cc:302] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418df36e55f740cb86162d1a96ac0926, 8efe3de199e241f6b5889e75f1856b1b; no voters:
I20260924 05:07:37.480108 17844 raft_consensus.cc:2781] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:37.480235 17844 raft_consensus.cc:686] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 1 LEADER]: Becoming Leader. State: Replica: 418df36e55f740cb86162d1a96ac0926, State: Running, Role: LEADER
I20260924 05:07:37.480310 17844 consensus_queue.cc:237] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } }
I20260924 05:07:37.481024 17365 catalog_manager.cc:5495] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418df36e55f740cb86162d1a96ac0926 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "418df36e55f740cb86162d1a96ac0926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418df36e55f740cb86162d1a96ac0926" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 } health_report { overall_health: UNKNOWN } } }
I20260924 05:07:37.536166 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.534180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.0.98.2:36601)
0924 05:07:37.534224 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.534226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.534234 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:37.534287 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.534288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.534307 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.534366 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:07:37.534452 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.534908 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.535589 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.535771 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.535812 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.535911 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:07:37.536007 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:07:37.536058 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.536070 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.536087 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:07:37.537942 17778 raft_consensus.cc:1261] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 1 FOLLOWER]: Refusing update from remote peer 418df36e55f740cb86162d1a96ac0926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:37.537941 17510 raft_consensus.cc:1261] T 0c474ddbecba4268ac8371bc9574f082 P 8efe3de199e241f6b5889e75f1856b1b [term 1 FOLLOWER]: Refusing update from remote peer 418df36e55f740cb86162d1a96ac0926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:37.538295 17844 consensus_queue.cc:1035] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe48104090ac4a3ca26cb7ed828afbca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:37.538439 17848 consensus_queue.cc:1035] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efe3de199e241f6b5889e75f1856b1b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:37.540998 17853 mvcc.cc:205] Tried to move back new op lower bound from 7332767570072657920 to 7332767569839906816. Current Snapshot: MvccSnapshot[applied={T|T < 7332767570072657920}]
I20260924 05:07:37.542622 17364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:37.542701 17364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:37.543377 17364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a648d427691f4533a4d7371b89706cb2: Sending DeleteTablet for 3 replicas of tablet 0c474ddbecba4268ac8371bc9574f082
I20260924 05:07:37.543735 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c474ddbecba4268ac8371bc9574f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40770
I20260924 05:07:37.543762 17758 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c474ddbecba4268ac8371bc9574f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35108
I20260924 05:07:37.544003 17624 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c474ddbecba4268ac8371bc9574f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260924 05:07:37.544040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17427
I20260924 05:07:37.545399 17862 tablet_replica.cc:330] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: stopping tablet replica
I20260924 05:07:37.545533 17862 raft_consensus.cc:2229] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:07:37.545691 17862 raft_consensus.cc:2258] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:37.546264 17862 ts_tablet_manager.cc:1905] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:37.546262 17861 tablet_replica.cc:330] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: stopping tablet replica
I20260924 05:07:37.546358 17861 raft_consensus.cc:2229] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:37.546459 17861 raft_consensus.cc:2258] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:37.547593 17862 ts_tablet_manager.cc:1918] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:37.547664 17862 log.cc:1196] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/0c474ddbecba4268ac8371bc9574f082
I20260924 05:07:37.547887 17862 ts_tablet_manager.cc:1939] T 0c474ddbecba4268ac8371bc9574f082 P 418df36e55f740cb86162d1a96ac0926: Deleting consensus metadata
I20260924 05:07:37.548209 17349 catalog_manager.cc:4841] TS 418df36e55f740cb86162d1a96ac0926 (127.0.98.2:36601): tablet 0c474ddbecba4268ac8371bc9574f082 (table test-table [id=d05973cce70842c3b8007accdb0b126b]) successfully deleted
W20260924 05:07:37.550107 17349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42189 (error 108)
I20260924 05:07:37.550297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17561
I20260924 05:07:37.550362 17861 ts_tablet_manager.cc:1905] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:37.551653 17861 ts_tablet_manager.cc:1918] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:37.551723 17861 log.cc:1196] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/0c474ddbecba4268ac8371bc9574f082
I20260924 05:07:37.551822 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.551710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42189 (local address 127.0.0.1:40774)
0924 05:07:37.551755 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.551791 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:07:37.551935 17349 catalog_manager.cc:4584] TS 8efe3de199e241f6b5889e75f1856b1b (127.0.98.1:42189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c474ddbecba4268ac8371bc9574f082: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42189: connect: Connection refused (error 111)
I20260924 05:07:37.552014 17861 ts_tablet_manager.cc:1939] T 0c474ddbecba4268ac8371bc9574f082 P fe48104090ac4a3ca26cb7ed828afbca: Deleting consensus metadata
I20260924 05:07:37.552366 17349 catalog_manager.cc:4841] TS fe48104090ac4a3ca26cb7ed828afbca (127.0.98.3:34635): tablet 0c474ddbecba4268ac8371bc9574f082 (table test-table [id=d05973cce70842c3b8007accdb0b126b]) successfully deleted
I20260924 05:07:37.555389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17695
I20260924 05:07:37.559512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17333
2026-09-24T05:07:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2044 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T05:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:37Z Disabled control of system clock
I20260924 05:07:37.583866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44663
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:37.663628 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:37.663836 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:37.663857 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:37.664956 17867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:37.664995 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:37.665006 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:37.665030 17867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:37.665051 17867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:37.666345 17867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:37.666558 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:37.666813 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:37.671031 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.671054 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.671054 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:37.671175 17867 server_base.cc:1034] running on GCE node
I20260924 05:07:37.671430 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:37.671653 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:37.672778 17867 hybrid_clock.cc:648] HybridClock initialized: now 1790226457672764 us; error 26 us; skew 500 ppm
I20260924 05:07:37.673827 17867 webserver.cc:460] Webserver started at http://127.0.98.62:41181/ using document root <none> and password file <none>
I20260924 05:07:37.674014 17867 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:37.674058 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:37.674171 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:37.675110 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "5bfb4063fbf448a2b780743d4f0583f9"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "44b66c7f7bac510a35362950b193d027"
server_key_iv: "72ad16b0773ba7ae0db6003de60a340f"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.675423 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bfb4063fbf448a2b780743d4f0583f9"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "44b66c7f7bac510a35362950b193d027"
server_key_iv: "72ad16b0773ba7ae0db6003de60a340f"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.676498 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:37.677131 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.677270 17867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:37.677343 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "5bfb4063fbf448a2b780743d4f0583f9"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "44b66c7f7bac510a35362950b193d027"
server_key_iv: "72ad16b0773ba7ae0db6003de60a340f"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.677399 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:37.688795 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:37.689034 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:37.689153 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:37.692520 17867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44663
I20260924 05:07:37.692549 17933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44663 every 8 connection(s)
I20260924 05:07:37.692826 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:37.693678 17934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:37.695748 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9: Bootstrap starting.
I20260924 05:07:37.696259 17934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:37.696508 17934 log.cc:824] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:37.697062 17934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9: No bootstrap required, opened a new log
I20260924 05:07:37.697306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17867
I20260924 05:07:37.697407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:37.697594 392 external_mini_cluster.cc:1428] Setting key 6e9c465551867b201f1c037a9bb9fa0d
I20260924 05:07:37.699010 17934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfb4063fbf448a2b780743d4f0583f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44663 } }
I20260924 05:07:37.699177 17934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:37.699213 17934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bfb4063fbf448a2b780743d4f0583f9, State: Initialized, Role: FOLLOWER
I20260924 05:07:37.699383 17934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfb4063fbf448a2b780743d4f0583f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44663 } }
I20260924 05:07:37.699470 17934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:37.699507 17934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:37.699553 17934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:37.700163 17934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfb4063fbf448a2b780743d4f0583f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44663 } }
I20260924 05:07:37.700264 17934 leader_election.cc:302] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bfb4063fbf448a2b780743d4f0583f9; no voters:
I20260924 05:07:37.700413 17934 leader_election.cc:288] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:37.700454 17939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:37.700580 17939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5bfb4063fbf448a2b780743d4f0583f9, State: Running, Role: LEADER
I20260924 05:07:37.700627 17934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:37.700702 17939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfb4063fbf448a2b780743d4f0583f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44663 } }
I20260924 05:07:37.701020 17939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bfb4063fbf448a2b780743d4f0583f9. Latest consensus state: current_term: 1 leader_uuid: "5bfb4063fbf448a2b780743d4f0583f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfb4063fbf448a2b780743d4f0583f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44663 } } }
I20260924 05:07:37.701228 17939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:37.701159 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bfb4063fbf448a2b780743d4f0583f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfb4063fbf448a2b780743d4f0583f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44663 } } }
I20260924 05:07:37.701597 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:37.701637 17948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:37.702200 17948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:37.703810 17948 catalog_manager.cc:1347] Generated new cluster ID: 8b5a1a4eab234370a66d9b7490b8472a
I20260924 05:07:37.703859 17948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:37.704281 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.698028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.98.62:44663)
0924 05:07:37.698237 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.698245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.698276 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:07:37.698982 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.698990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.699064 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.699210 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:37.699390 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.701038 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.702170 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.702735 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.702845 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.703440 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:37.703444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:37.703444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:37.703627 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:37.703667 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:37.703681 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:37.703744 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.703999 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.704074 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 05:07:37.738660 17948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:37.739415 17948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:37.743187 17948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9: Generated new TSK 0
I20260924 05:07:37.743376 17948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:37.744678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44663
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:37.821197 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:37.821411 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:37.821442 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:37.822644 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:37.822686 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:37.822719 17958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:37.824270 17958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:37.824519 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:37.824810 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:37.829437 17964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.829463 17963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.829461 17966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:37.829703 17958 server_base.cc:1034] running on GCE node
I20260924 05:07:37.830006 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:37.830252 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:37.831389 17958 hybrid_clock.cc:648] HybridClock initialized: now 1790226457831378 us; error 30 us; skew 500 ppm
I20260924 05:07:37.832603 17958 webserver.cc:460] Webserver started at http://127.0.98.1:36615/ using document root <none> and password file <none>
I20260924 05:07:37.832829 17958 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:37.832878 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:37.833019 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:37.834154 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a1970392e1f458290cde370a68a25af"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "2a1605b57b4462079a2587fefbc18f8a"
server_key_iv: "0bbd06e9f8e97d0922f55af73499e712"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.834483 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a1970392e1f458290cde370a68a25af"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "2a1605b57b4462079a2587fefbc18f8a"
server_key_iv: "0bbd06e9f8e97d0922f55af73499e712"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.835666 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:37.836337 17972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.836488 17958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:37.836582 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "8a1970392e1f458290cde370a68a25af"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "2a1605b57b4462079a2587fefbc18f8a"
server_key_iv: "0bbd06e9f8e97d0922f55af73499e712"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.836642 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:37.849051 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:37.849292 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:37.849401 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:37.849587 17958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:37.849925 17958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:37.849964 17958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.849992 17958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:37.850009 17958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.855058 17958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44463
I20260924 05:07:37.855115 18085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44463 every 8 connection(s)
I20260924 05:07:37.855371 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:07:37.858750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 17958
I20260924 05:07:37.858846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:07:37.859045 392 external_mini_cluster.cc:1428] Setting key 003c2f9f516e482db00fadd4d1eba5a0
I20260924 05:07:37.859858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44663
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:07:37.860302 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.855972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58239 (local address 127.0.98.62:44663)
0924 05:07:37.856013 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.856015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.856238 (+ 223us) server_negotiation.cc:407] Connection header received
0924 05:07:37.856710 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.856711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.856729 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.856779 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:37.857411 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.857944 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.858886 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.859185 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.859258 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.860056 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:37.860057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:37.860057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:37.860062 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:37.860078 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:37.860083 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:37.860128 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.860198 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.860227 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:07:37.860437 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.855962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44663 (local address 127.0.98.1:58239)
0924 05:07:37.856133 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.856163 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.856622 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.856854 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.856858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.856868 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:37.857329 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.857333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.858025 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.858026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.858808 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.858810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.859266 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.859268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.859406 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.859917 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:37.859918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:37.859946 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:37.859974 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:37.860140 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:37.860142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:37.860145 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.860185 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.860263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260924 05:07:37.861121 18086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44663
I20260924 05:07:37.861217 18086 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:37.861395 18086 heartbeater.cc:511] Master 127.0.98.62:44663 requested a full tablet report, sending...
I20260924 05:07:37.861887 17898 ts_manager.cc:194] Registered new tserver with Master: 8a1970392e1f458290cde370a68a25af (127.0.98.1:44463)
I20260924 05:07:37.862455 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58239
W20260924 05:07:37.937829 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:37.938053 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:37.938086 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:37.939324 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:37.939363 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:37.939396 18089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:37.940766 18089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:37.940999 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:37.941267 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:37.945773 18094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.945768 18095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:37.945766 18097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:37.945868 18089 server_base.cc:1034] running on GCE node
I20260924 05:07:37.946130 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:37.946347 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:37.947489 18089 hybrid_clock.cc:648] HybridClock initialized: now 1790226457947471 us; error 32 us; skew 500 ppm
I20260924 05:07:37.948647 18089 webserver.cc:460] Webserver started at http://127.0.98.2:37407/ using document root <none> and password file <none>
I20260924 05:07:37.948854 18089 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:37.948901 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:37.949023 18089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:37.950039 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "eafab38d30824c8d829f0c95953d1967"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "c9fa7f88d25ec1dee085f2b6fdd30e09"
server_key_iv: "24f5a4df35ec0dcf96b40a29b23887ba"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.950347 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eafab38d30824c8d829f0c95953d1967"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "c9fa7f88d25ec1dee085f2b6fdd30e09"
server_key_iv: "24f5a4df35ec0dcf96b40a29b23887ba"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.951483 18089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:37.952075 18103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.952225 18089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:37.952309 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "eafab38d30824c8d829f0c95953d1967"
format_stamp: "Formatted at 2026-09-24 05:07:37 on dist-test-slave-m01t"
server_key: "c9fa7f88d25ec1dee085f2b6fdd30e09"
server_key_iv: "24f5a4df35ec0dcf96b40a29b23887ba"
server_key_version: "encryptionkey@0"
I20260924 05:07:37.952368 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:37.969162 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:37.969545 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:37.969681 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:37.969899 18089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:37.970201 18089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:37.970233 18089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.970261 18089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:37.970278 18089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:37.975646 18089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35885
I20260924 05:07:37.975684 18216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35885 every 8 connection(s)
I20260924 05:07:37.975970 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:37.980906 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.976673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57399 (local address 127.0.98.62:44663)
0924 05:07:37.976742 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:37.976744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:37.976936 (+ 192us) server_negotiation.cc:407] Connection header received
0924 05:07:37.977435 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:37.977436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:37.977445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:37.977525 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:37.978257 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.978761 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.979675 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.979892 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.979965 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.980703 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:37.980704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:37.980704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:37.980707 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:37.980717 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:37.980720 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:37.980755 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:37.980811 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 05:07:37.980830 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:07:37.981058 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:37.976638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44663 (local address 127.0.98.2:57399)
0924 05:07:37.976815 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:37.976853 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:37.977343 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:37.977578 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:37.977582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:37.977589 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:37.978175 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.978181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.978831 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.978832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.979599 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:37.979601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:37.979973 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:37.979973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:37.980105 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:37.980588 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:37.980588 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:37.980614 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:37.980635 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:37.980757 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:37.980759 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:37.980761 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:07:37.980802 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:07:37.980873 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260924 05:07:37.981487 18217 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44663
I20260924 05:07:37.981576 18217 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:37.981762 18217 heartbeater.cc:511] Master 127.0.98.62:44663 requested a full tablet report, sending...
I20260924 05:07:37.982116 17898 ts_manager.cc:194] Registered new tserver with Master: eafab38d30824c8d829f0c95953d1967 (127.0.98.2:35885)
I20260924 05:07:37.982447 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57399
I20260924 05:07:37.984527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18089
I20260924 05:07:37.984623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:07:37.984800 392 external_mini_cluster.cc:1428] Setting key e3d055a2f874ebf4caafd89cd7f92423
I20260924 05:07:37.985690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44663
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:38.065028 18220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:38.065228 18220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:38.065258 18220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:38.066413 18220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:38.066450 18220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:38.066483 18220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:38.067759 18220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:38.067975 18220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:38.068240 18220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:38.072763 18226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:38.072803 18225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:38.072927 18220 server_base.cc:1034] running on GCE node
W20260924 05:07:38.072786 18228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:38.073179 18220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:38.073426 18220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:38.074563 18220 hybrid_clock.cc:648] HybridClock initialized: now 1790226458074543 us; error 32 us; skew 500 ppm
I20260924 05:07:38.075706 18220 webserver.cc:460] Webserver started at http://127.0.98.3:39023/ using document root <none> and password file <none>
I20260924 05:07:38.075919 18220 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:38.075965 18220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:38.076087 18220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:38.077056 18220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd4ae680c7224bf5b218426db10534f9"
format_stamp: "Formatted at 2026-09-24 05:07:38 on dist-test-slave-m01t"
server_key: "54c6dfeac10dfbcc554014b153f7d06a"
server_key_iv: "ac9709eca40a2ae57bbe0b9a6e5df6e1"
server_key_version: "encryptionkey@0"
I20260924 05:07:38.077382 18220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd4ae680c7224bf5b218426db10534f9"
format_stamp: "Formatted at 2026-09-24 05:07:38 on dist-test-slave-m01t"
server_key: "54c6dfeac10dfbcc554014b153f7d06a"
server_key_iv: "ac9709eca40a2ae57bbe0b9a6e5df6e1"
server_key_version: "encryptionkey@0"
I20260924 05:07:38.078608 18220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:38.079259 18234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:38.079406 18220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:38.079481 18220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "fd4ae680c7224bf5b218426db10534f9"
format_stamp: "Formatted at 2026-09-24 05:07:38 on dist-test-slave-m01t"
server_key: "54c6dfeac10dfbcc554014b153f7d06a"
server_key_iv: "ac9709eca40a2ae57bbe0b9a6e5df6e1"
server_key_version: "encryptionkey@0"
I20260924 05:07:38.079550 18220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:38.089733 18220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:38.089987 18220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:38.090106 18220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:38.090305 18220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:38.090590 18220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:38.090622 18220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:38.090648 18220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:38.090665 18220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:38.095890 18220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34367
I20260924 05:07:38.095956 18347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34367 every 8 connection(s)
I20260924 05:07:38.096227 18220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:07:38.099671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18220
I20260924 05:07:38.099781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:07:38.099995 392 external_mini_cluster.cc:1428] Setting key 7eecf5c0eb27d1e67f6a3e9b79ddfa40
I20260924 05:07:38.101120 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.096837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60811 (local address 127.0.98.62:44663)
0924 05:07:38.096881 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.096883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.097109 (+ 226us) server_negotiation.cc:407] Connection header received
0924 05:07:38.097560 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.097561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.097571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.097633 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:38.098284 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.098762 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.099643 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.099936 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.099999 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.100849 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:38.100850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:38.100850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:38.100854 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:38.100869 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:38.100874 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:38.100921 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.101034 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.101059 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:07:38.101312 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.096793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44663 (local address 127.0.98.3:60811)
0924 05:07:38.096992 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.097030 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:38.097468 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:38.097726 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:38.097730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:38.097739 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:38.098216 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.098221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.098829 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.098830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.099565 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.099567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.100016 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.100017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.100169 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.100701 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:38.100702 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:38.100737 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:38.100763 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:38.100951 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:38.100956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:38.100959 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:38.101037 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:07:38.101117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260924 05:07:38.101727 18348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44663
I20260924 05:07:38.101812 18348 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:38.101943 18348 heartbeater.cc:511] Master 127.0.98.62:44663 requested a full tablet report, sending...
I20260924 05:07:38.102236 17898 ts_manager.cc:194] Registered new tserver with Master: fd4ae680c7224bf5b218426db10534f9 (127.0.98.3:34367)
I20260924 05:07:38.102579 17898 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60811
I20260924 05:07:38.111404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:07:38.115382 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.112686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.62:44663)
0924 05:07:38.112758 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.112760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.112828 (+ 68us) server_negotiation.cc:407] Connection header received
0924 05:07:38.112878 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.112879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.112891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.112936 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:38.113100 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.113558 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.114334 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.114514 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.114565 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.115165 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:38.115166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:38.115166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:38.115170 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:38.115184 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:38.115188 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:38.115230 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.115284 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.115306 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 05:07:38.116869 17898 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:07:38.118485 17898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:07:38.124503 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.120437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35885 (local address 127.0.0.1:55038)
0924 05:07:38.120618 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.120657 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:38.120720 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:38.120938 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:38.120941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:38.120943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:38.121006 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.121008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.122317 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.122318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.123507 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.123508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.124204 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.124206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.124284 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.124298 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:07:38.124363 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 05:07:38.124420 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":46,"thread_start_us":57,"threads_started":1}
I20260924 05:07:38.124513 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.120529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.0.98.2:35885)
0924 05:07:38.120693 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.120698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.120710 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:07:38.120840 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.120843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.120855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.120918 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:38.121077 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.122231 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.123617 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.124107 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.124176 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.124272 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.124391 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.124429 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260924 05:07:38.124616 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.120437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44463 (local address 127.0.0.1:46712)
0924 05:07:38.120783 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.120794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:38.120845 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:38.121110 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:38.121110 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:38.121112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:38.121141 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.121142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.122171 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.122172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.123500 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.123502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.124407 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.124408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.124468 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.124472 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:38.124500 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 05:07:38.124541 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":21,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260924 05:07:38.124890 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.120644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34367 (local address 127.0.0.1:35280)
0924 05:07:38.121242 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.121251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:38.121321 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:38.121541 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:38.121542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:38.121544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:38.121582 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.121584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.122668 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.122669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.123516 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.123518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.124489 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.124490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.124540 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.124543 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:38.124789 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 05:07:38.124830 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260924 05:07:38.125030 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.120815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.0.98.1:44463)
0924 05:07:38.120957 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.120962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.120971 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:38.121008 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.121010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.121025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.121111 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:38.121198 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.122093 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.123594 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.124312 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.124792 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.124888 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.124914 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.124952 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260924 05:07:38.125135 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.121000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.0.98.3:34367)
0924 05:07:38.121192 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.121196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.121297 (+ 101us) server_negotiation.cc:407] Connection header received
0924 05:07:38.121403 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.121405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.121419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.121467 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:38.121661 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.122592 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.123596 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.124312 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.124376 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.124461 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.125020 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.125062 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260924 05:07:38.125275 18151 tablet_service.cc:1467] Processing CreateTablet for tablet 42227f4dbac5424faaa67cc99bf74a14 (DEFAULT_TABLE table=test-table [id=b309c6d82ba54aeeb0b4fcdcc09e7397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:38.125669 18151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42227f4dbac5424faaa67cc99bf74a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:38.125788 18020 tablet_service.cc:1467] Processing CreateTablet for tablet 42227f4dbac5424faaa67cc99bf74a14 (DEFAULT_TABLE table=test-table [id=b309c6d82ba54aeeb0b4fcdcc09e7397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:38.125887 18282 tablet_service.cc:1467] Processing CreateTablet for tablet 42227f4dbac5424faaa67cc99bf74a14 (DEFAULT_TABLE table=test-table [id=b309c6d82ba54aeeb0b4fcdcc09e7397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:38.126109 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42227f4dbac5424faaa67cc99bf74a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:38.126202 18282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42227f4dbac5424faaa67cc99bf74a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:38.128644 18362 tablet_bootstrap.cc:492] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af: Bootstrap starting.
I20260924 05:07:38.129051 18363 tablet_bootstrap.cc:492] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Bootstrap starting.
I20260924 05:07:38.129346 18362 tablet_bootstrap.cc:654] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:38.129549 18364 tablet_bootstrap.cc:492] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Bootstrap starting.
I20260924 05:07:38.129618 18362 log.cc:824] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:38.129675 18363 tablet_bootstrap.cc:654] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:38.130002 18363 log.cc:824] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:38.130169 18364 tablet_bootstrap.cc:654] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:38.130366 18362 tablet_bootstrap.cc:492] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af: No bootstrap required, opened a new log
I20260924 05:07:38.130412 18364 log.cc:824] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:38.130441 18362 ts_tablet_manager.cc:1397] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:38.130757 18363 tablet_bootstrap.cc:492] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: No bootstrap required, opened a new log
I20260924 05:07:38.130815 18363 ts_tablet_manager.cc:1397] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:07:38.131156 18364 tablet_bootstrap.cc:492] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: No bootstrap required, opened a new log
I20260924 05:07:38.131222 18364 ts_tablet_manager.cc:1397] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:38.132531 18362 raft_consensus.cc:348] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.132668 18362 raft_consensus.cc:374] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:38.132699 18362 raft_consensus.cc:729] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1970392e1f458290cde370a68a25af, State: Initialized, Role: FOLLOWER
I20260924 05:07:38.132987 18362 consensus_queue.cc:260] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.133248 18362 ts_tablet_manager.cc:1428] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:07:38.133229 18363 raft_consensus.cc:348] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.133356 18086 heartbeater.cc:503] Master 127.0.98.62:44663 was elected leader, sending a full tablet report...
I20260924 05:07:38.133390 18363 raft_consensus.cc:374] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:38.133428 18363 raft_consensus.cc:729] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafab38d30824c8d829f0c95953d1967, State: Initialized, Role: FOLLOWER
I20260924 05:07:38.133541 18363 consensus_queue.cc:260] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.133576 18364 raft_consensus.cc:348] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.133731 18364 raft_consensus.cc:374] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:38.133776 18364 raft_consensus.cc:729] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4ae680c7224bf5b218426db10534f9, State: Initialized, Role: FOLLOWER
I20260924 05:07:38.133821 18363 ts_tablet_manager.cc:1428] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:07:38.133879 18364 consensus_queue.cc:260] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.133952 18217 heartbeater.cc:503] Master 127.0.98.62:44663 was elected leader, sending a full tablet report...
I20260924 05:07:38.134101 18364 ts_tablet_manager.cc:1428] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:07:38.134241 18348 heartbeater.cc:503] Master 127.0.98.62:44663 was elected leader, sending a full tablet report...
I20260924 05:07:38.162964 18369 raft_consensus.cc:482] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:38.163080 18369 raft_consensus.cc:504] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.163394 18369 leader_election.cc:288] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a1970392e1f458290cde370a68a25af (127.0.98.1:44463), fd4ae680c7224bf5b218426db10534f9 (127.0.98.3:34367)
I20260924 05:07:38.166221 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.163564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44463 (local address 127.0.98.2:44205)
0924 05:07:38.163625 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.163637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:38.163680 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:38.163857 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:38.163858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:38.163860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:38.163919 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.163921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.164528 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.164529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.165427 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.165429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.165976 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.165977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.166047 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.166051 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:38.166095 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:07:38.166137 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 05:07:38.166208 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.163637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44205 (local address 127.0.98.1:44463)
0924 05:07:38.163693 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.163695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.163705 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:07:38.163737 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.163738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.163750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.163808 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:38.163995 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.164453 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.165504 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.165900 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.165963 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.165973 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.166109 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.166140 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:07:38.166455 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.163758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51243 (local address 127.0.98.3:34367)
0924 05:07:38.163832 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.163834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.163915 (+ 81us) server_negotiation.cc:407] Connection header received
0924 05:07:38.163962 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.163963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.163971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.164014 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:38.164172 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.164636 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.165753 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.166096 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.166161 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.166171 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.166352 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.166396 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 05:07:38.166455 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.163666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34367 (local address 127.0.98.2:51243)
0924 05:07:38.163847 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.163856 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:38.163905 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:38.164035 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:38.164036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:38.164037 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:38.164106 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.164107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.164720 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.164721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.165663 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:38.165666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.166186 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.166187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:38.166258 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.166261 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:38.166338 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:07:38.166380 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260924 05:07:38.166533 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42227f4dbac5424faaa67cc99bf74a14" candidate_uuid: "eafab38d30824c8d829f0c95953d1967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1970392e1f458290cde370a68a25af" is_pre_election: true
I20260924 05:07:38.166675 18040 raft_consensus.cc:2454] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafab38d30824c8d829f0c95953d1967 in term 0.
I20260924 05:07:38.166909 18106 leader_election.cc:302] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1970392e1f458290cde370a68a25af, eafab38d30824c8d829f0c95953d1967; no voters:
I20260924 05:07:38.167037 18369 raft_consensus.cc:2781] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:07:38.167083 18369 raft_consensus.cc:482] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:07:38.167099 18369 raft_consensus.cc:3037] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:38.167042 18302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42227f4dbac5424faaa67cc99bf74a14" candidate_uuid: "eafab38d30824c8d829f0c95953d1967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd4ae680c7224bf5b218426db10534f9" is_pre_election: true
I20260924 05:07:38.167179 18302 raft_consensus.cc:2454] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafab38d30824c8d829f0c95953d1967 in term 0.
I20260924 05:07:38.167774 18369 raft_consensus.cc:504] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.167876 18369 leader_election.cc:288] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [CANDIDATE]: Term 1 election: Requested vote from peers 8a1970392e1f458290cde370a68a25af (127.0.98.1:44463), fd4ae680c7224bf5b218426db10534f9 (127.0.98.3:34367)
I20260924 05:07:38.168036 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42227f4dbac5424faaa67cc99bf74a14" candidate_uuid: "eafab38d30824c8d829f0c95953d1967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1970392e1f458290cde370a68a25af"
I20260924 05:07:38.168113 18040 raft_consensus.cc:3037] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:38.168124 18302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42227f4dbac5424faaa67cc99bf74a14" candidate_uuid: "eafab38d30824c8d829f0c95953d1967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd4ae680c7224bf5b218426db10534f9"
I20260924 05:07:38.168200 18302 raft_consensus.cc:3037] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:38.168783 18040 raft_consensus.cc:2454] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafab38d30824c8d829f0c95953d1967 in term 1.
I20260924 05:07:38.168885 18302 raft_consensus.cc:2454] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafab38d30824c8d829f0c95953d1967 in term 1.
I20260924 05:07:38.168946 18106 leader_election.cc:302] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1970392e1f458290cde370a68a25af, eafab38d30824c8d829f0c95953d1967; no voters:
I20260924 05:07:38.169024 18369 raft_consensus.cc:2781] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:38.169153 18369 raft_consensus.cc:686] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 1 LEADER]: Becoming Leader. State: Replica: eafab38d30824c8d829f0c95953d1967, State: Running, Role: LEADER
I20260924 05:07:38.169236 18369 consensus_queue.cc:237] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } }
I20260924 05:07:38.169986 17898 catalog_manager.cc:5495] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 reported cstate change: term changed from 0 to 1, leader changed from <none> to eafab38d30824c8d829f0c95953d1967 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eafab38d30824c8d829f0c95953d1967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafab38d30824c8d829f0c95953d1967" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260924 05:07:38.181026 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.178797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.0.98.2:35885)
0924 05:07:38.178848 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:38.178849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:38.178858 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:38.178887 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:38.178888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:38.178903 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:38.178964 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:07:38.179076 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.179569 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.180317 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:38.180527 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:38.180581 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:38.180717 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:07:38.180814 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:07:38.180884 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:38.180931 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:07:38.180946 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:07:38.182680 18302 raft_consensus.cc:1261] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 1 FOLLOWER]: Refusing update from remote peer eafab38d30824c8d829f0c95953d1967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:38.182680 18040 raft_consensus.cc:1261] T 42227f4dbac5424faaa67cc99bf74a14 P 8a1970392e1f458290cde370a68a25af [term 1 FOLLOWER]: Refusing update from remote peer eafab38d30824c8d829f0c95953d1967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:38.183007 18372 consensus_queue.cc:1035] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4ae680c7224bf5b218426db10534f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:38.183123 18369 consensus_queue.cc:1035] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1970392e1f458290cde370a68a25af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:38.185257 18375 mvcc.cc:205] Tried to move back new op lower bound from 7332767572713758720 to 7332767572661784576. Current Snapshot: MvccSnapshot[applied={T|T < 7332767572713758720}]
I20260924 05:07:38.185360 18377 mvcc.cc:205] Tried to move back new op lower bound from 7332767572713758720 to 7332767572661784576. Current Snapshot: MvccSnapshot[applied={T|T < 7332767572713758720}]
I20260924 05:07:38.185366 18376 mvcc.cc:205] Tried to move back new op lower bound from 7332767572713758720 to 7332767572661784576. Current Snapshot: MvccSnapshot[applied={T|T < 7332767572713758720}]
I20260924 05:07:38.187453 17897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:38.187536 17897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:38.188235 17897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bfb4063fbf448a2b780743d4f0583f9: Sending DeleteTablet for 3 replicas of tablet 42227f4dbac5424faaa67cc99bf74a14
I20260924 05:07:38.188548 18282 tablet_service.cc:1514] Processing DeleteTablet for tablet 42227f4dbac5424faaa67cc99bf74a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:38 UTC) from {username='slave'} at 127.0.0.1:35280
I20260924 05:07:38.188540 18020 tablet_service.cc:1514] Processing DeleteTablet for tablet 42227f4dbac5424faaa67cc99bf74a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:38 UTC) from {username='slave'} at 127.0.0.1:46712
I20260924 05:07:38.188546 18151 tablet_service.cc:1514] Processing DeleteTablet for tablet 42227f4dbac5424faaa67cc99bf74a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:38 UTC) from {username='slave'} at 127.0.0.1:55038
I20260924 05:07:38.188759 18385 tablet_replica.cc:330] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: stopping tablet replica
I20260924 05:07:38.188915 18385 raft_consensus.cc:2229] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:38.189013 18385 raft_consensus.cc:2258] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:38.189069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17958
I20260924 05:07:38.189875 18385 ts_tablet_manager.cc:1905] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:38.190158 18387 tablet_replica.cc:330] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: stopping tablet replica
I20260924 05:07:38.190240 18387 raft_consensus.cc:2229] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:07:38.190339 18387 raft_consensus.cc:2258] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:38.190737 18387 ts_tablet_manager.cc:1905] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:38.191414 18385 ts_tablet_manager.cc:1918] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:38.191471 18385 log.cc:1196] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/42227f4dbac5424faaa67cc99bf74a14
I20260924 05:07:38.191713 18385 ts_tablet_manager.cc:1939] T 42227f4dbac5424faaa67cc99bf74a14 P fd4ae680c7224bf5b218426db10534f9: Deleting consensus metadata
I20260924 05:07:38.192230 18387 ts_tablet_manager.cc:1918] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:38.192256 17884 catalog_manager.cc:4841] TS fd4ae680c7224bf5b218426db10534f9 (127.0.98.3:34367): tablet 42227f4dbac5424faaa67cc99bf74a14 (table test-table [id=b309c6d82ba54aeeb0b4fcdcc09e7397]) successfully deleted
I20260924 05:07:38.192289 18387 log.cc:1196] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/42227f4dbac5424faaa67cc99bf74a14
I20260924 05:07:38.192504 18387 ts_tablet_manager.cc:1939] T 42227f4dbac5424faaa67cc99bf74a14 P eafab38d30824c8d829f0c95953d1967: Deleting consensus metadata
I20260924 05:07:38.192894 17883 catalog_manager.cc:4841] TS eafab38d30824c8d829f0c95953d1967 (127.0.98.2:35885): tablet 42227f4dbac5424faaa67cc99bf74a14 (table test-table [id=b309c6d82ba54aeeb0b4fcdcc09e7397]) successfully deleted
W20260924 05:07:38.195106 17884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44463 (error 108)
I20260924 05:07:38.195313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18089
I20260924 05:07:38.195851 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:38.195458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44463 (local address 127.0.0.1:46728)
0924 05:07:38.195773 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:38.195820 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
W20260924 05:07:38.196658 17884 catalog_manager.cc:4584] TS 8a1970392e1f458290cde370a68a25af (127.0.98.1:44463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42227f4dbac5424faaa67cc99bf74a14: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44463: connect: Connection refused (error 111)
I20260924 05:07:38.199678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18220
I20260924 05:07:38.203222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 17867
2026-09-24T05:07:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (642 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:38 dist-test-slave-m01t krb5kdc[18391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:38 dist-test-slave-m01t krb5kdc[18391](info): set up 2 sockets
Sep 24 05:07:38 dist-test-slave-m01t krb5kdc[18391](info): commencing operation
krb5kdc: starting...
W20260924 05:07:39.145159 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:07:39.250655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:39.328404 18404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:39.328624 18404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:39.328646 18404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:39.329905 18404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:39.329952 18404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:39.329965 18404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:39.329977 18404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:39.329989 18404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:39.331537 18404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:39.331776 18404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:39.332002 18404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:39.336725 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:39.336737 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:39.336751 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:39.336817 18404 server_base.cc:1034] running on GCE node
I20260924 05:07:39.337167 18404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:39.337446 18404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:39.338600 18404 hybrid_clock.cc:648] HybridClock initialized: now 1790226459338586 us; error 29 us; skew 500 ppm
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:39.345877 18404 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:07:39.346288 18404 webserver.cc:460] Webserver started at http://127.0.98.62:38583/ using document root <none> and password file <none>
I20260924 05:07:39.346478 18404 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:39.346524 18404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:39.346642 18404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:39.347648 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "f964ac3d0ad268e34fd0c0ad67d21ec7"
server_key_iv: "e1019cbafb75e9aa9c3df5e131a2513f"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.347970 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "f964ac3d0ad268e34fd0c0ad67d21ec7"
server_key_iv: "e1019cbafb75e9aa9c3df5e131a2513f"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.349129 18404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:39.349784 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.349921 18404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:39.349999 18404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "f964ac3d0ad268e34fd0c0ad67d21ec7"
server_key_iv: "e1019cbafb75e9aa9c3df5e131a2513f"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.350063 18404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:39.362442 18404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:39.364498 18404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:39.364617 18404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:39.368182 18404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40979
I20260924 05:07:39.368211 18471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40979 every 8 connection(s)
I20260924 05:07:39.368505 18404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:39.369237 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:39.371382 18472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d: Bootstrap starting.
I20260924 05:07:39.371907 18472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:39.372149 18472 log.cc:824] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:39.372745 18472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d: No bootstrap required, opened a new log
I20260924 05:07:39.374287 18472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260924 05:07:39.374415 18472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:39.374437 18472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1e61b6d20b4d6c8942e7d5eb95480d, State: Initialized, Role: FOLLOWER
I20260924 05:07:39.374568 18472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260924 05:07:39.374630 18472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:39.374612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18404
I20260924 05:07:39.374650 18472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:39.374676 18472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:39.374712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:39.374843 392 external_mini_cluster.cc:1428] Setting key d34e861720f842c965faea874df834ed
I20260924 05:07:39.375164 18472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260924 05:07:39.375269 18472 leader_election.cc:302] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b1e61b6d20b4d6c8942e7d5eb95480d; no voters:
I20260924 05:07:39.375412 18472 leader_election.cc:288] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:39.375447 18477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:39.375576 18477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [term 1 LEADER]: Becoming Leader. State: Replica: 1b1e61b6d20b4d6c8942e7d5eb95480d, State: Running, Role: LEADER
I20260924 05:07:39.375655 18472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:39.375653 18477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } }
I20260924 05:07:39.376150 18478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } } }
I20260924 05:07:39.376236 18478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:39.376528 18479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b1e61b6d20b4d6c8942e7d5eb95480d. Latest consensus state: current_term: 1 leader_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1e61b6d20b4d6c8942e7d5eb95480d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40979 } } }
I20260924 05:07:39.376614 18479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:39.376690 18486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:39.377161 18486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:39.378656 18486 catalog_manager.cc:1347] Generated new cluster ID: 0e3aabd35ea34547bd1e8962e056a34f
I20260924 05:07:39.378703 18486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:39.386536 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.375258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.0.98.62:40979)
0924 05:07:39.375422 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.375431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.375461 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:07:39.376070 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.376078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.376139 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.376924 (+ 785us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:39.377047 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.378342 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.379356 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.379942 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.380035 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.382588 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:39.382592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:39.382592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:39.382602 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:39.385022 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.385376 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.385379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.385379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.385492 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.385677 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.385678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.385678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.385995 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:39.386143 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.386211 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.386336 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260924 05:07:39.402972 18486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:39.403668 18486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:39.406196 18486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d: Generated new TSK 0
I20260924 05:07:39.406354 18486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:07:39.432266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40979
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:39.510062 18498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:39.510242 18498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:39.510272 18498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:39.511368 18498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:39.511406 18498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:39.511440 18498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:39.512840 18498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:39.513044 18498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:39.513242 18498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:39.517762 18503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:39.517773 18504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:39.517923 18498 server_base.cc:1034] running on GCE node
W20260924 05:07:39.517787 18506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:39.518152 18498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:39.518391 18498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:39.519534 18498 hybrid_clock.cc:648] HybridClock initialized: now 1790226459519513 us; error 32 us; skew 500 ppm
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:39.521821 18498 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:07:39.522255 18498 webserver.cc:460] Webserver started at http://127.0.98.1:34967/ using document root <none> and password file <none>
I20260924 05:07:39.522449 18498 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:39.522495 18498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:39.522624 18498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:39.523633 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "19ae398ce0d845bfb2387b01f9a2d049"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "4a7a646d8507f4bd7b5c9c78372be6bf"
server_key_iv: "b45245a42ea57257602da99aa26ce23d"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.523958 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "19ae398ce0d845bfb2387b01f9a2d049"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "4a7a646d8507f4bd7b5c9c78372be6bf"
server_key_iv: "b45245a42ea57257602da99aa26ce23d"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.525154 18498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:39.525830 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.525988 18498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.526067 18498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "19ae398ce0d845bfb2387b01f9a2d049"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "4a7a646d8507f4bd7b5c9c78372be6bf"
server_key_iv: "b45245a42ea57257602da99aa26ce23d"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.526137 18498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:39.554419 18498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:39.556203 18498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:39.556335 18498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:39.556553 18498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:39.556854 18498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:39.556890 18498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.556917 18498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:39.556936 18498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.562299 18498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36941
I20260924 05:07:39.562359 18626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36941 every 8 connection(s)
I20260924 05:07:39.562644 18498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:07:39.567263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18498
I20260924 05:07:39.567369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:07:39.567595 392 external_mini_cluster.cc:1428] Setting key 60504e47af2dde975176b6521d01cc95
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:39.571635 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.563261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34993 (local address 127.0.98.62:40979)
0924 05:07:39.563325 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.563327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.563505 (+ 178us) server_negotiation.cc:407] Connection header received
0924 05:07:39.564438 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.564439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.564452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.564501 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:39.565057 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.565665 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.566640 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.566835 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.566890 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.569069 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:39.569070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:39.569070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:39.569076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:39.570449 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.570800 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.570801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.570802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.570859 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.571052 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.571053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.571053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.571172 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:39.571234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.571447 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.571528 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:07:39.571692 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.563225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40979 (local address 127.0.98.1:34993)
0924 05:07:39.563381 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:39.563422 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:39.564351 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:39.564561 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:39.564564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:39.564814 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:39.564992 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.564996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.565783 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.565784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.566562 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.566565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.566903 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.566904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.566994 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.567441 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:39.567450 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:39.568982 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:39.570524 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:39.570530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:39.570531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:39.570729 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:39.570913 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:39.570914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:39.570914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:39.570999 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:39.571226 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:39.571229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:39.571328 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 05:07:39.571399 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:07:39.571540 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260924 05:07:39.572103 18627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40979
I20260924 05:07:39.572197 18627 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:39.572381 18627 heartbeater.cc:511] Master 127.0.98.62:40979 requested a full tablet report, sending...
I20260924 05:07:39.572834 18436 ts_manager.cc:194] Registered new tserver with Master: 19ae398ce0d845bfb2387b01f9a2d049 (127.0.98.1:36941)
I20260924 05:07:39.573393 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:07:39.590005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40979
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:39.667384 18632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:39.667560 18632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:39.667582 18632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:39.668648 18632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:39.668694 18632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:39.668727 18632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:39.670145 18632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:39.670307 18632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:39.670467 18632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:39.674844 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:39.674865 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:39.675038 18632 server_base.cc:1034] running on GCE node
W20260924 05:07:39.674844 18640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:39.675310 18632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:39.675498 18632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:39.676623 18632 hybrid_clock.cc:648] HybridClock initialized: now 1790226459676612 us; error 25 us; skew 500 ppm
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:39.678917 18632 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:07:39.679280 18632 webserver.cc:460] Webserver started at http://127.0.98.2:33607/ using document root <none> and password file <none>
I20260924 05:07:39.679469 18632 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:39.679514 18632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:39.679625 18632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:39.680516 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b45f9c9a2c44388a93dbe3a383d4549"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "ff434d14894b1d3ad05b3edfb14496bc"
server_key_iv: "756e68949f2edc7b567a3a628f3dfc81"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.680810 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b45f9c9a2c44388a93dbe3a383d4549"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "ff434d14894b1d3ad05b3edfb14496bc"
server_key_iv: "756e68949f2edc7b567a3a628f3dfc81"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.681945 18632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:39.682543 18647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.682693 18632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:39.682768 18632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "9b45f9c9a2c44388a93dbe3a383d4549"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "ff434d14894b1d3ad05b3edfb14496bc"
server_key_iv: "756e68949f2edc7b567a3a628f3dfc81"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.682825 18632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:39.691644 18632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:39.693956 18632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:39.694101 18632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:39.694286 18632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:39.694559 18632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:39.694593 18632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.694619 18632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:39.694636 18632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.699738 18632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36335
I20260924 05:07:39.699792 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36335 every 8 connection(s)
I20260924 05:07:39.700058 18632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:39.703631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18632
I20260924 05:07:39.703728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:07:39.703907 392 external_mini_cluster.cc:1428] Setting key d569673ea3613710fa7114f59b6ebc96
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:39.708752 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.700686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47051 (local address 127.0.98.62:40979)
0924 05:07:39.700745 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.700746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.700911 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:07:39.701748 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.701749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.701759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.701808 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:39.702333 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.702809 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.703753 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.704028 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.704098 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.706286 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:39.706287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:39.706288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:39.706294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:39.707550 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.707914 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.707915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.707915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.707967 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.708187 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.708188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.708188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.708289 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:39.708343 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.708587 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.708667 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:07:39.708899 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.700654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40979 (local address 127.0.98.2:47051)
0924 05:07:39.700800 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:39.700838 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:39.701649 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:39.701869 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:39.701872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:39.702101 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:39.702271 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.702275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.702875 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.702875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.703655 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.703657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.704118 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.704119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.704248 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.704700 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:39.704710 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:39.706195 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:39.707640 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:39.707647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:39.707649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:39.707840 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:39.708027 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:39.708028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:39.708028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:39.708122 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:39.708352 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:39.708358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:39.708468 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 05:07:39.708551 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:07:39.708699 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260924 05:07:39.709571 18761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40979
I20260924 05:07:39.709689 18761 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:39.709946 18761 heartbeater.cc:511] Master 127.0.98.62:40979 requested a full tablet report, sending...
I20260924 05:07:39.710453 18436 ts_manager.cc:194] Registered new tserver with Master: 9b45f9c9a2c44388a93dbe3a383d4549 (127.0.98.2:36335)
I20260924 05:07:39.710944 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47051
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:07:39.727897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40979
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:39.806196 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:39.806382 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:39.806406 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:39.807538 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:39.807578 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:39.807605 18766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:39.809077 18766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:39.809293 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:39.809510 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:07:39.814188 18766 server_base.cc:1034] running on GCE node
W20260924 05:07:39.814179 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:39.814247 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:39.814177 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:39.814466 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:39.814688 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:39.815831 18766 hybrid_clock.cc:648] HybridClock initialized: now 1790226459815816 us; error 27 us; skew 500 ppm
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:39.818181 18766 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:07:39.818593 18766 webserver.cc:460] Webserver started at http://127.0.98.3:43515/ using document root <none> and password file <none>
I20260924 05:07:39.818804 18766 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:39.818849 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:39.818967 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:39.819941 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "f87699e1a9bc4b69b99c8217c5543a59"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "702e626427c3bebd4497573feb6466c9"
server_key_iv: "4a9c9aca20cb8b9794f0da72d5d6c90c"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.820250 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f87699e1a9bc4b69b99c8217c5543a59"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "702e626427c3bebd4497573feb6466c9"
server_key_iv: "4a9c9aca20cb8b9794f0da72d5d6c90c"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.821421 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:07:39.822166 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.822329 18766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.822404 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "f87699e1a9bc4b69b99c8217c5543a59"
format_stamp: "Formatted at 2026-09-24 05:07:39 on dist-test-slave-m01t"
server_key: "702e626427c3bebd4497573feb6466c9"
server_key_iv: "4a9c9aca20cb8b9794f0da72d5d6c90c"
server_key_version: "encryptionkey@0"
I20260924 05:07:39.822466 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:39.835748 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:39.837963 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:39.838104 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:39.838322 18766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:39.838616 18766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:39.838649 18766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.838677 18766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:39.838706 18766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:39.844066 18766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40575
I20260924 05:07:39.844131 18894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40575 every 8 connection(s)
I20260924 05:07:39.844403 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:07:39 dist-test-slave-m01t krb5kdc[18391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:39.852272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18766
I20260924 05:07:39.852388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:07:39.852567 392 external_mini_cluster.cc:1428] Setting key 5a04484e0de994976ebd7d15c14e4ce3
I20260924 05:07:39.852818 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.845002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56607 (local address 127.0.98.62:40979)
0924 05:07:39.845062 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.845064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.845237 (+ 173us) server_negotiation.cc:407] Connection header received
0924 05:07:39.846195 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.846197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.846209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.846259 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:39.846868 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.847362 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.848358 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.848544 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.848600 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.850541 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:39.850542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:39.850542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:39.850549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:39.851561 (+ 1012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.851921 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.851922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.851922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.851965 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.852190 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.852190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.852191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.852328 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:39.852395 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.852665 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.852739 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:07:39.852974 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.844961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40979 (local address 127.0.98.3:56607)
0924 05:07:39.845118 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:39.845158 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:39.846066 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:39.846330 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:39.846334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:39.846576 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:39.846761 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.846765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.847452 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.847453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.848257 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.848260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.848619 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.848621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.848739 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.849138 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:39.849146 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:39.850456 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:39.851629 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:39.851633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:39.851635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:39.851863 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:39.852012 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:39.852012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:39.852013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:39.852127 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:39.852395 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:39.852400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:39.852528 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 05:07:39.852616 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:07:39.852777 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260924 05:07:39.853338 18895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40979
I20260924 05:07:39.853421 18895 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:39.853577 18895 heartbeater.cc:511] Master 127.0.98.62:40979 requested a full tablet report, sending...
I20260924 05:07:39.853941 18435 ts_manager.cc:194] Registered new tserver with Master: f87699e1a9bc4b69b99c8217c5543a59 (127.0.98.3:40575)
I20260924 05:07:39.854270 18435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56607
I20260924 05:07:39.864171 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:07:39.870632 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.865463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.0.98.62:40979)
0924 05:07:39.865517 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.865519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.865585 (+ 66us) server_negotiation.cc:407] Connection header received
0924 05:07:39.865633 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.865634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.865641 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.865683 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:39.866198 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.866640 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.867288 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.867449 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.867488 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.868517 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:39.868518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:39.868519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:39.868524 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:39.869657 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.870052 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.870053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.870053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.870099 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:39.870266 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:39.870267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:39.870267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:39.870372 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:39.870444 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.870525 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.870566 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:07:39.872133 18435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:07:39.873618 18435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:07:39.879558 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.875504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40575 (local address 127.0.0.1:60430)
0924 05:07:39.875835 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:39.875870 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:39.875937 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:39.876140 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:39.876142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:39.876144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:39.876209 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.876211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.877239 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.877240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.878456 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.878459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.879292 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.879293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.879366 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.879377 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 05:07:39.879442 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 05:07:39.879490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":45,"threads_started":1}
I20260924 05:07:39.879690 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.875665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.0.98.1:36941)
0924 05:07:39.875796 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.875801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.876060 (+ 259us) server_negotiation.cc:407] Connection header received
0924 05:07:39.876136 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.876138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.876153 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.876240 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:39.876455 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.877587 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.878768 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.879287 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.879352 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.879407 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.879576 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.879617 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260924 05:07:39.879879 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.875667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36335 (local address 127.0.0.1:44070)
0924 05:07:39.876038 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:39.876046 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:39.876087 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:39.876331 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:39.876332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:39.876335 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:39.876379 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.876379 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.877742 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.877743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.878758 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.878761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.879647 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.879648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.879716 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.879720 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:39.879762 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:07:39.879802 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":23,"thread_start_us":53,"threads_started":1}
I20260924 05:07:39.879897 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.875560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.98.3:40575)
0924 05:07:39.875773 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.875778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.875941 (+ 163us) server_negotiation.cc:407] Connection header received
0924 05:07:39.876018 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.876021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.876035 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.876136 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:39.876285 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.877148 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.878538 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.879205 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.879273 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.879325 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.879469 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.879545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260924 05:07:39.879881 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.875513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36941 (local address 127.0.0.1:58716)
0924 05:07:39.875991 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:39.876000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:39.876059 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:39.876330 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:39.876331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:39.876333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:39.876378 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.876379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.877706 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.877707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.878680 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:39.878682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.879438 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.879438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:39.879491 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.879495 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:39.879765 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 05:07:39.879813 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":37,"thread_start_us":36,"threads_started":1}
I20260924 05:07:39.880342 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:39.875849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.98.2:36335)
0924 05:07:39.876010 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:39.876014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:39.876135 (+ 121us) server_negotiation.cc:407] Connection header received
0924 05:07:39.876171 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:39.876175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:39.876188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:39.876249 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:39.876448 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.877587 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.878818 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:39.879504 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:39.879563 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:39.879612 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:39.880224 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 05:07:39.880270 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260924 05:07:39.880537 18829 tablet_service.cc:1467] Processing CreateTablet for tablet 2adc2684a1564752955d2914f3249a3d (DEFAULT_TABLE table=test-table [id=1df4b65872ef4d9fb3ebeda5c350767a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:39.880566 18561 tablet_service.cc:1467] Processing CreateTablet for tablet 2adc2684a1564752955d2914f3249a3d (DEFAULT_TABLE table=test-table [id=1df4b65872ef4d9fb3ebeda5c350767a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:39.880837 18829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adc2684a1564752955d2914f3249a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:39.880897 18561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adc2684a1564752955d2914f3249a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:39.880861 18695 tablet_service.cc:1467] Processing CreateTablet for tablet 2adc2684a1564752955d2914f3249a3d (DEFAULT_TABLE table=test-table [id=1df4b65872ef4d9fb3ebeda5c350767a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:39.881147 18695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2adc2684a1564752955d2914f3249a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:39.883209 18909 tablet_bootstrap.cc:492] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Bootstrap starting.
I20260924 05:07:39.883740 18909 tablet_bootstrap.cc:654] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:39.883978 18911 tablet_bootstrap.cc:492] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Bootstrap starting.
I20260924 05:07:39.884241 18909 log.cc:824] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:39.884678 18911 tablet_bootstrap.cc:654] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:39.884963 18909 tablet_bootstrap.cc:492] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: No bootstrap required, opened a new log
I20260924 05:07:39.885036 18909 ts_tablet_manager.cc:1397] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:39.885264 18911 log.cc:824] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:39.885596 18910 tablet_bootstrap.cc:492] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049: Bootstrap starting.
I20260924 05:07:39.885946 18911 tablet_bootstrap.cc:492] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: No bootstrap required, opened a new log
I20260924 05:07:39.886005 18911 ts_tablet_manager.cc:1397] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:39.886240 18910 tablet_bootstrap.cc:654] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:39.886731 18910 log.cc:824] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:39.886955 18909 raft_consensus.cc:348] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:39.887125 18909 raft_consensus.cc:374] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:39.887162 18909 raft_consensus.cc:729] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87699e1a9bc4b69b99c8217c5543a59, State: Initialized, Role: FOLLOWER
I20260924 05:07:39.887286 18909 consensus_queue.cc:260] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:39.887534 18909 ts_tablet_manager.cc:1428] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:07:39.887552 18895 heartbeater.cc:503] Master 127.0.98.62:40979 was elected leader, sending a full tablet report...
I20260924 05:07:39.887835 18911 raft_consensus.cc:348] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:39.888001 18911 raft_consensus.cc:374] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:39.888034 18911 raft_consensus.cc:729] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b45f9c9a2c44388a93dbe3a383d4549, State: Initialized, Role: FOLLOWER
I20260924 05:07:39.888072 18910 tablet_bootstrap.cc:492] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049: No bootstrap required, opened a new log
I20260924 05:07:39.888115 18910 ts_tablet_manager.cc:1397] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 05:07:39.888137 18911 consensus_queue.cc:260] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:39.888401 18911 ts_tablet_manager.cc:1428] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:07:39.888418 18761 heartbeater.cc:503] Master 127.0.98.62:40979 was elected leader, sending a full tablet report...
I20260924 05:07:39.889814 18910 raft_consensus.cc:348] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:39.889936 18910 raft_consensus.cc:374] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:39.889956 18910 raft_consensus.cc:729] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ae398ce0d845bfb2387b01f9a2d049, State: Initialized, Role: FOLLOWER
I20260924 05:07:39.890034 18910 consensus_queue.cc:260] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:39.890247 18910 ts_tablet_manager.cc:1428] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:39.890288 18627 heartbeater.cc:503] Master 127.0.98.62:40979 was elected leader, sending a full tablet report...
W20260924 05:07:39.950556 18762 tablet.cc:2369] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:07:40.004580 18917 raft_consensus.cc:482] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:40.004701 18917 raft_consensus.cc:504] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:40.005059 18917 leader_election.cc:288] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f87699e1a9bc4b69b99c8217c5543a59 (127.0.98.3:40575), 9b45f9c9a2c44388a93dbe3a383d4549 (127.0.98.2:36335)
I20260924 05:07:40.008615 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.005274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35561 (local address 127.0.98.3:40575)
0924 05:07:40.005317 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.005319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.005328 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:40.005379 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.005379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.005388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.005437 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.005585 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.006359 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.007711 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.008269 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.008338 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.008347 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.008503 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.008541 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:07:40.008612 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.005372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59185 (local address 127.0.98.2:36335)
0924 05:07:40.005416 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.005418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.005512 (+ 94us) server_negotiation.cc:407] Connection header received
0924 05:07:40.005568 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.005570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.005582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.005634 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.005758 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.006431 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.007709 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.008269 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.008338 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.008347 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.008503 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.008541 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:07:40.008620 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.005195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40575 (local address 127.0.98.1:35561)
0924 05:07:40.005250 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.005262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.005316 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.005448 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.005449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.005452 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.005517 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.005519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.006431 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.006432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.007626 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.007628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.008349 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.008350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.008437 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.008440 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.008484 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.008528 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":6}
I20260924 05:07:40.008625 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.005290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36335 (local address 127.0.98.1:59185)
0924 05:07:40.005446 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.005454 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.005510 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.005633 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.005634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.005635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.005689 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.005690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.006499 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.006499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.007628 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.007629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.008349 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.008350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.008426 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.008430 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.008484 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.008528 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":11,"thread_start_us":95,"threads_started":1}
I20260924 05:07:40.008997 18849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adc2684a1564752955d2914f3249a3d" candidate_uuid: "19ae398ce0d845bfb2387b01f9a2d049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87699e1a9bc4b69b99c8217c5543a59" is_pre_election: true
I20260924 05:07:40.009006 18715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adc2684a1564752955d2914f3249a3d" candidate_uuid: "19ae398ce0d845bfb2387b01f9a2d049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" is_pre_election: true
I20260924 05:07:40.009130 18715 raft_consensus.cc:2454] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ae398ce0d845bfb2387b01f9a2d049 in term 0.
I20260924 05:07:40.009131 18849 raft_consensus.cc:2454] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ae398ce0d845bfb2387b01f9a2d049 in term 0.
I20260924 05:07:40.009351 18516 leader_election.cc:302] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ae398ce0d845bfb2387b01f9a2d049, f87699e1a9bc4b69b99c8217c5543a59; no voters:
I20260924 05:07:40.009474 18917 raft_consensus.cc:2781] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:07:40.009536 18917 raft_consensus.cc:482] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:07:40.009558 18917 raft_consensus.cc:3037] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.010191 18917 raft_consensus.cc:504] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:40.010296 18917 leader_election.cc:288] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [CANDIDATE]: Term 1 election: Requested vote from peers f87699e1a9bc4b69b99c8217c5543a59 (127.0.98.3:40575), 9b45f9c9a2c44388a93dbe3a383d4549 (127.0.98.2:36335)
I20260924 05:07:40.010453 18849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adc2684a1564752955d2914f3249a3d" candidate_uuid: "19ae398ce0d845bfb2387b01f9a2d049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87699e1a9bc4b69b99c8217c5543a59"
I20260924 05:07:40.010468 18715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2adc2684a1564752955d2914f3249a3d" candidate_uuid: "19ae398ce0d845bfb2387b01f9a2d049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b45f9c9a2c44388a93dbe3a383d4549"
I20260924 05:07:40.010531 18715 raft_consensus.cc:3037] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.010531 18849 raft_consensus.cc:3037] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.011336 18715 raft_consensus.cc:2454] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ae398ce0d845bfb2387b01f9a2d049 in term 1.
I20260924 05:07:40.011336 18849 raft_consensus.cc:2454] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ae398ce0d845bfb2387b01f9a2d049 in term 1.
I20260924 05:07:40.011513 18516 leader_election.cc:302] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ae398ce0d845bfb2387b01f9a2d049, f87699e1a9bc4b69b99c8217c5543a59; no voters:
I20260924 05:07:40.011584 18917 raft_consensus.cc:2781] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:40.011742 18917 raft_consensus.cc:686] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [term 1 LEADER]: Becoming Leader. State: Replica: 19ae398ce0d845bfb2387b01f9a2d049, State: Running, Role: LEADER
I20260924 05:07:40.011823 18917 consensus_queue.cc:237] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } }
I20260924 05:07:40.012579 18435 catalog_manager.cc:5495] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19ae398ce0d845bfb2387b01f9a2d049 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "19ae398ce0d845bfb2387b01f9a2d049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19ae398ce0d845bfb2387b01f9a2d049" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36941 } health_report { overall_health: HEALTHY } } }
I20260924 05:07:40.025378 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.023346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.0.98.1:36941)
0924 05:07:40.023396 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.023398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.023406 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:40.023437 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.023437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.023453 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.023515 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:07:40.023628 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.024076 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.024749 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.024933 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.024980 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.025123 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:07:40.025208 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:07:40.025273 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.025292 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.025302 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:07:40.027148 18849 raft_consensus.cc:1261] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 1 FOLLOWER]: Refusing update from remote peer 19ae398ce0d845bfb2387b01f9a2d049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:40.027148 18715 raft_consensus.cc:1261] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 1 FOLLOWER]: Refusing update from remote peer 19ae398ce0d845bfb2387b01f9a2d049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:40.027484 18917 consensus_queue.cc:1035] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b45f9c9a2c44388a93dbe3a383d4549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:40.027635 18919 consensus_queue.cc:1035] T 2adc2684a1564752955d2914f3249a3d P 19ae398ce0d845bfb2387b01f9a2d049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87699e1a9bc4b69b99c8217c5543a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:40.030340 18922 mvcc.cc:205] Tried to move back new op lower bound from 7332767580268363776 to 7332767580208889856. Current Snapshot: MvccSnapshot[applied={T|T < 7332767580268363776 or (T in {7332767580268363776})}]
I20260924 05:07:40.031566 18924 mvcc.cc:205] Tried to move back new op lower bound from 7332767580268363776 to 7332767580208889856. Current Snapshot: MvccSnapshot[applied={T|T < 7332767580268363776 or (T in {7332767580268363776})}]
I20260924 05:07:40.032327 18435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:40.032423 18435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:40.033128 18435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b1e61b6d20b4d6c8942e7d5eb95480d: Sending DeleteTablet for 3 replicas of tablet 2adc2684a1564752955d2914f3249a3d
I20260924 05:07:40.033427 18561 tablet_service.cc:1514] Processing DeleteTablet for tablet 2adc2684a1564752955d2914f3249a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58716
I20260924 05:07:40.033509 18695 tablet_service.cc:1514] Processing DeleteTablet for tablet 2adc2684a1564752955d2914f3249a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44070
I20260924 05:07:40.033511 18829 tablet_service.cc:1514] Processing DeleteTablet for tablet 2adc2684a1564752955d2914f3249a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60430
I20260924 05:07:40.033747 18934 tablet_replica.cc:330] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: stopping tablet replica
I20260924 05:07:40.033850 18934 raft_consensus.cc:2229] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:40.033846 18933 tablet_replica.cc:330] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: stopping tablet replica
I20260924 05:07:40.033869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18498
I20260924 05:07:40.033952 18934 raft_consensus.cc:2258] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:40.034580 18934 ts_tablet_manager.cc:1905] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:40.035846 18934 ts_tablet_manager.cc:1918] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:40.035892 18934 log.cc:1196] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/2adc2684a1564752955d2914f3249a3d
I20260924 05:07:40.036058 18934 ts_tablet_manager.cc:1939] T 2adc2684a1564752955d2914f3249a3d P f87699e1a9bc4b69b99c8217c5543a59: Deleting consensus metadata
I20260924 05:07:40.036448 18422 catalog_manager.cc:4841] TS f87699e1a9bc4b69b99c8217c5543a59 (127.0.98.3:40575): tablet 2adc2684a1564752955d2914f3249a3d (table test-table [id=1df4b65872ef4d9fb3ebeda5c350767a]) successfully deleted
I20260924 05:07:40.036721 18933 raft_consensus.cc:2229] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:40.036839 18933 raft_consensus.cc:2258] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:40.037189 18933 ts_tablet_manager.cc:1905] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:40.038628 18933 ts_tablet_manager.cc:1918] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:40.038707 18933 log.cc:1196] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/2adc2684a1564752955d2914f3249a3d
I20260924 05:07:40.038947 18933 ts_tablet_manager.cc:1939] T 2adc2684a1564752955d2914f3249a3d P 9b45f9c9a2c44388a93dbe3a383d4549: Deleting consensus metadata
W20260924 05:07:40.039270 18423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36941 (error 108)
I20260924 05:07:40.039276 18422 catalog_manager.cc:4841] TS 9b45f9c9a2c44388a93dbe3a383d4549 (127.0.98.2:36335): tablet 2adc2684a1564752955d2914f3249a3d (table test-table [id=1df4b65872ef4d9fb3ebeda5c350767a]) successfully deleted
I20260924 05:07:40.039669 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.039563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36941 (local address 127.0.0.1:58740)
0924 05:07:40.039613 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.039645 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:07:40.039772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18632
W20260924 05:07:40.039776 18423 catalog_manager.cc:4584] TS 19ae398ce0d845bfb2387b01f9a2d049 (127.0.98.1:36941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2adc2684a1564752955d2914f3249a3d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36941: connect: Connection refused (error 111)
I20260924 05:07:40.044252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18766
I20260924 05:07:40.047679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18404
2026-09-24T05:07:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1844 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T05:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:40Z Disabled control of system clock
I20260924 05:07:40.071885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35219
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:40.148802 18939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:40.148995 18939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:40.149014 18939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:40.150141 18939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:40.150179 18939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:40.150190 18939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:40.150202 18939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:40.150211 18939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:40.151496 18939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:40.151716 18939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:40.151973 18939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:40.156329 18944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:40.156450 18939 server_base.cc:1034] running on GCE node
W20260924 05:07:40.156343 18945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:40.156365 18947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:40.156826 18939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:40.157083 18939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:40.158221 18939 hybrid_clock.cc:648] HybridClock initialized: now 1790226460158205 us; error 28 us; skew 500 ppm
I20260924 05:07:40.159276 18939 webserver.cc:460] Webserver started at http://127.0.98.62:43933/ using document root <none> and password file <none>
I20260924 05:07:40.159467 18939 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:40.159513 18939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:40.159629 18939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:40.160549 18939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "6dcd9461fa144cc0b98d39efc714e870"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "b4585fb83bde4647403d1155064a7cf3"
server_key_iv: "3b7f7ddc0347817285afa609e0ef7640"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.160856 18939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "6dcd9461fa144cc0b98d39efc714e870"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "b4585fb83bde4647403d1155064a7cf3"
server_key_iv: "3b7f7ddc0347817285afa609e0ef7640"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.162001 18939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:07:40.162629 18953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.162775 18939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:40.162850 18939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "6dcd9461fa144cc0b98d39efc714e870"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "b4585fb83bde4647403d1155064a7cf3"
server_key_iv: "3b7f7ddc0347817285afa609e0ef7640"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.162910 18939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:40.177819 18939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:40.178063 18939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:40.178186 18939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:40.181445 18939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35219
I20260924 05:07:40.181479 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35219 every 8 connection(s)
I20260924 05:07:40.181779 18939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:40.182642 19006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:40.184664 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870: Bootstrap starting.
I20260924 05:07:40.185151 19006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:40.185385 19006 log.cc:824] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:40.185491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 18939
I20260924 05:07:40.185590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:40.185810 392 external_mini_cluster.cc:1428] Setting key 9e72759211f46c6d6a173b7f2c6056d9
I20260924 05:07:40.186118 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870: No bootstrap required, opened a new log
I20260924 05:07:40.188413 19006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcd9461fa144cc0b98d39efc714e870" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35219 } }
I20260924 05:07:40.188545 19006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:40.188568 19006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcd9461fa144cc0b98d39efc714e870, State: Initialized, Role: FOLLOWER
I20260924 05:07:40.188685 19006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcd9461fa144cc0b98d39efc714e870" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35219 } }
I20260924 05:07:40.188766 19006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:40.188793 19006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:40.188838 19006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.189494 19006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcd9461fa144cc0b98d39efc714e870" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35219 } }
I20260924 05:07:40.189602 19006 leader_election.cc:302] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dcd9461fa144cc0b98d39efc714e870; no voters:
I20260924 05:07:40.189801 19006 leader_election.cc:288] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:40.189823 19011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:40.189929 19011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [term 1 LEADER]: Becoming Leader. State: Replica: 6dcd9461fa144cc0b98d39efc714e870, State: Running, Role: LEADER
I20260924 05:07:40.190060 19006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:40.190066 19011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcd9461fa144cc0b98d39efc714e870" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35219 } }
I20260924 05:07:40.190510 19013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dcd9461fa144cc0b98d39efc714e870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcd9461fa144cc0b98d39efc714e870" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35219 } } }
I20260924 05:07:40.190745 19013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:40.190675 19012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcd9461fa144cc0b98d39efc714e870. Latest consensus state: current_term: 1 leader_uuid: "6dcd9461fa144cc0b98d39efc714e870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcd9461fa144cc0b98d39efc714e870" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35219 } } }
I20260924 05:07:40.190932 19012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:40.191560 19020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:40.191811 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.186256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.98.62:35219)
0924 05:07:40.186445 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.186452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.186474 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:07:40.187258 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.187265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.187319 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.187426 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:40.187678 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.188521 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.189594 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.190217 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.190322 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.191017 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:40.191021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:40.191021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:40.191226 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:40.191268 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:40.191284 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:40.191516 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.191555 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.191632 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260924 05:07:40.192183 19020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:40.193565 19020 catalog_manager.cc:1347] Generated new cluster ID: 7601d9994648420aa2392c9356636d4e
I20260924 05:07:40.193619 19020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:40.210937 19020 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:40.211411 19020 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:40.220474 19020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870: Generated new TSK 0
I20260924 05:07:40.220674 19020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:40.226713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35219
--builtin_ntp_servers=127.0.98.20:41919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:40.309126 19030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:40.309301 19030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:40.309320 19030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:40.310439 19030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:40.310475 19030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:40.310498 19030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:40.311828 19030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:40.312032 19030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:40.312300 19030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:40.316784 19035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:40.316833 19038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:40.316802 19036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:40.317093 19030 server_base.cc:1034] running on GCE node
I20260924 05:07:40.317291 19030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:40.317529 19030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:40.318666 19030 hybrid_clock.cc:648] HybridClock initialized: now 1790226460318636 us; error 49 us; skew 500 ppm
I20260924 05:07:40.319809 19030 webserver.cc:460] Webserver started at http://127.0.98.1:37005/ using document root <none> and password file <none>
I20260924 05:07:40.320032 19030 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:40.320075 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:40.320207 19030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:40.321225 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b35a42c61de4435b6e5983d6dda450e"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "6d60860717b8d0520c4273cc0ee50131"
server_key_iv: "a7cbc5ac37680f983cc05fed21927a21"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.321563 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b35a42c61de4435b6e5983d6dda450e"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "6d60860717b8d0520c4273cc0ee50131"
server_key_iv: "a7cbc5ac37680f983cc05fed21927a21"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.322772 19030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:07:40.323422 19044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.323585 19030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:40.323663 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "4b35a42c61de4435b6e5983d6dda450e"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "6d60860717b8d0520c4273cc0ee50131"
server_key_iv: "a7cbc5ac37680f983cc05fed21927a21"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.323730 19030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:40.343626 19030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:40.343931 19030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:40.344067 19030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:40.344291 19030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:40.344600 19030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:40.344633 19030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.344661 19030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:40.344677 19030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.349676 19030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43825
I20260924 05:07:40.349776 19157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43825 every 8 connection(s)
I20260924 05:07:40.350031 19030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:07:40.351941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19030
I20260924 05:07:40.352056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:07:40.352241 392 external_mini_cluster.cc:1428] Setting key 474aac2d3d92fa78266859e624cf2b1b
I20260924 05:07:40.353205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35219
--builtin_ntp_servers=127.0.98.20:41919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:07:40.355345 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.350683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37861 (local address 127.0.98.62:35219)
0924 05:07:40.350750 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.350752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.350948 (+ 196us) server_negotiation.cc:407] Connection header received
0924 05:07:40.351408 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.351409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.351421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.351476 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:40.352224 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.352736 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.354041 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.354230 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.354284 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.355100 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:40.355101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:40.355101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:40.355106 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:40.355121 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:40.355125 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:40.355184 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.355252 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.355273 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 05:07:40.355484 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.350653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35219 (local address 127.0.98.1:37861)
0924 05:07:40.350833 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.350872 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.351337 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.351528 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.351531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.351538 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:40.352135 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.352143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.352810 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.352811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.353969 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.353972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.354312 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.354313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.354468 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.354966 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:40.354966 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:40.354997 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:40.355019 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:40.355186 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:40.355189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:40.355191 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.355234 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.355306 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260924 05:07:40.356086 19158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35219
I20260924 05:07:40.356174 19158 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:40.356343 19158 heartbeater.cc:511] Master 127.0.98.62:35219 requested a full tablet report, sending...
I20260924 05:07:40.356829 18970 ts_manager.cc:194] Registered new tserver with Master: 4b35a42c61de4435b6e5983d6dda450e (127.0.98.1:43825)
I20260924 05:07:40.357399 18970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37861
W20260924 05:07:40.430500 19161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:40.430716 19161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:40.430738 19161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:40.431927 19161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:40.431973 19161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:40.432000 19161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:40.433377 19161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:40.433625 19161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:40.433967 19161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:40.438699 19167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:40.438675 19169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:40.438669 19166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:40.438805 19161 server_base.cc:1034] running on GCE node
I20260924 05:07:40.439090 19161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:40.439329 19161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:40.440481 19161 hybrid_clock.cc:648] HybridClock initialized: now 1790226460440469 us; error 23 us; skew 500 ppm
I20260924 05:07:40.441754 19161 webserver.cc:460] Webserver started at http://127.0.98.2:42531/ using document root <none> and password file <none>
I20260924 05:07:40.441994 19161 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:40.442044 19161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:40.442173 19161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:40.443210 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "529ebaf649ee4a10bbd088b388bd3e59"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "10732432c201e009315f8189faaba87e"
server_key_iv: "6fdb09d4bc4aea3a8ca5c3b6ca2eebbf"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.443531 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "529ebaf649ee4a10bbd088b388bd3e59"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "10732432c201e009315f8189faaba87e"
server_key_iv: "6fdb09d4bc4aea3a8ca5c3b6ca2eebbf"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.444671 19161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:40.445279 19175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.445438 19161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:40.445513 19161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "529ebaf649ee4a10bbd088b388bd3e59"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "10732432c201e009315f8189faaba87e"
server_key_iv: "6fdb09d4bc4aea3a8ca5c3b6ca2eebbf"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.445580 19161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:40.461596 19161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:40.461869 19161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:40.461989 19161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:40.462181 19161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:40.462486 19161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:40.462519 19161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.462545 19161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:40.462560 19161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.467835 19161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39447
I20260924 05:07:40.467903 19288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39447 every 8 connection(s)
I20260924 05:07:40.468165 19161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:40.472757 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.468843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53869 (local address 127.0.98.62:35219)
0924 05:07:40.468909 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.468911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.469087 (+ 176us) server_negotiation.cc:407] Connection header received
0924 05:07:40.469533 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.469534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.469542 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.469594 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:40.470284 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.470758 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.471660 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.471836 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.471881 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.472583 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:40.472584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:40.472584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:40.472587 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:40.472596 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:40.472599 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:40.472627 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.472686 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.472702 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:07:40.472947 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.468807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35219 (local address 127.0.98.2:53869)
0924 05:07:40.468971 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.469009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.469456 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.469654 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.469657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.469664 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:40.470209 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.470213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.470828 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.470829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.471585 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.471586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.471897 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.471898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.471994 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.472473 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:40.472473 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:40.472500 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:40.472521 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:40.472634 (+ 113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:40.472636 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:40.472639 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.472680 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.472751 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 05:07:40.473371 19289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35219
I20260924 05:07:40.473446 19289 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:40.473616 19289 heartbeater.cc:511] Master 127.0.98.62:35219 requested a full tablet report, sending...
I20260924 05:07:40.473970 18970 ts_manager.cc:194] Registered new tserver with Master: 529ebaf649ee4a10bbd088b388bd3e59 (127.0.98.2:39447)
I20260924 05:07:40.474298 18970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53869
I20260924 05:07:40.477664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19161
I20260924 05:07:40.477784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:07:40.477942 392 external_mini_cluster.cc:1428] Setting key 3a590e18e82bca231b75aba3d0818254
I20260924 05:07:40.478706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35219
--builtin_ntp_servers=127.0.98.20:41919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:40.557379 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:40.557602 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:40.557623 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:40.558738 19292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:40.558777 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:40.558800 19292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:40.560078 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:40.560271 19292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:40.560504 19292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:40.565090 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:40.565117 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:40.565263 19292 server_base.cc:1034] running on GCE node
W20260924 05:07:40.565356 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:40.565600 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:40.565913 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:40.567059 19292 hybrid_clock.cc:648] HybridClock initialized: now 1790226460567010 us; error 59 us; skew 500 ppm
I20260924 05:07:40.568296 19292 webserver.cc:460] Webserver started at http://127.0.98.3:45061/ using document root <none> and password file <none>
I20260924 05:07:40.568524 19292 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:40.568574 19292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:40.568699 19292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:40.569774 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "73394ea54aed464ebbd935a8255ff0b7"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "1f479364ba308553b09631fb496b6a22"
server_key_iv: "1c680b7694ef9e9ab79c3b93b8e64870"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.570128 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73394ea54aed464ebbd935a8255ff0b7"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "1f479364ba308553b09631fb496b6a22"
server_key_iv: "1c680b7694ef9e9ab79c3b93b8e64870"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.571275 19292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:07:40.571902 19306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.572067 19292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:40.572154 19292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "73394ea54aed464ebbd935a8255ff0b7"
format_stamp: "Formatted at 2026-09-24 05:07:40 on dist-test-slave-m01t"
server_key: "1f479364ba308553b09631fb496b6a22"
server_key_iv: "1c680b7694ef9e9ab79c3b93b8e64870"
server_key_version: "encryptionkey@0"
I20260924 05:07:40.572219 19292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:40.585242 19292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:40.585520 19292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:40.585647 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:40.585868 19292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:40.586196 19292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:40.586289 19292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.586360 19292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:40.586386 19292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:40.592124 19292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43677
I20260924 05:07:40.592182 19419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43677 every 8 connection(s)
I20260924 05:07:40.592486 19292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:07:40.592909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19292
I20260924 05:07:40.593011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:07:40.593194 392 external_mini_cluster.cc:1428] Setting key 356db94e901aaf799abc1bd163414008
I20260924 05:07:40.597430 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.593395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46521 (local address 127.0.98.62:35219)
0924 05:07:40.593490 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.593491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.593498 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:07:40.594113 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.594114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.594122 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.594153 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:40.594775 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.595266 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.596235 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.596407 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.596449 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.597263 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:40.597264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:40.597264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:40.597267 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:40.597276 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:40.597279 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:40.597310 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.597356 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.597372 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 05:07:40.597661 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.593131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35219 (local address 127.0.98.3:46521)
0924 05:07:40.593333 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.593381 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.594014 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.594208 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.594212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.594220 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:40.594712 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.594716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.595351 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.595352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.596168 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.596171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.596467 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.596468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.596597 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.597148 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:40.597148 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:40.597177 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:40.597200 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:40.597312 (+ 112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:40.597314 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:40.597316 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.597359 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.597436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 05:07:40.598117 19420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35219
I20260924 05:07:40.598205 19420 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:40.598369 19420 heartbeater.cc:511] Master 127.0.98.62:35219 requested a full tablet report, sending...
I20260924 05:07:40.598706 18970 ts_manager.cc:194] Registered new tserver with Master: 73394ea54aed464ebbd935a8255ff0b7 (127.0.98.3:43677)
I20260924 05:07:40.599052 18970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46521
I20260924 05:07:40.604657 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:07:40.608676 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.605939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.62:35219)
0924 05:07:40.605998 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.606000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.606063 (+ 63us) server_negotiation.cc:407] Connection header received
0924 05:07:40.606128 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.606129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.606138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.606189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:40.606327 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.606786 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.607515 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.607715 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.607766 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.608400 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:40.608401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:40.608401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:40.608406 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:40.608421 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:40.608424 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:40.608504 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.608579 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.608589 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:07:40.610252 18970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:07:40.611763 18970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:07:40.617589 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.613698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43677 (local address 127.0.0.1:53972)
0924 05:07:40.614029 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.614037 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.614067 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.614622 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.614623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.614625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.614660 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.614661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.615820 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.615821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.616600 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.616602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.617284 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.617285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.617362 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.617376 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.617440 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.617512 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":116,"threads_started":1}
I20260924 05:07:40.617600 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.614245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.0.98.3:43677)
0924 05:07:40.614381 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.614385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.614393 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:40.614425 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.614427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.614440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.614518 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.614735 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.615739 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.616681 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.617190 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.617255 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.617349 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.617453 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.617512 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260924 05:07:40.618011 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.613679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39447 (local address 127.0.0.1:52758)
0924 05:07:40.613858 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.613881 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.613940 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.614427 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.614428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.614430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.614476 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.614478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.615436 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.615438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.616685 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.616686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.617743 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.617743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.617816 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.617820 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.617874 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.617913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":12,"thread_start_us":65,"threads_started":1}
I20260924 05:07:40.618043 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.614063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.98.2:39447)
0924 05:07:40.614253 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.614257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.614267 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:07:40.614301 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.614304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.614318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.614422 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.614556 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.615350 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.616782 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.617527 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.617595 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.617684 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.617917 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.617962 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260924 05:07:40.618010 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.613679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43825 (local address 127.0.0.1:59026)
0924 05:07:40.613844 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.613880 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.613940 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.614252 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.614255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.614257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.614321 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.614322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.615623 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.615624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.616683 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.616686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.617741 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.617742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.617816 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.617820 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.617870 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.617913 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":5,"thread_start_us":129,"threads_started":1}
I20260924 05:07:40.618362 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.613890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.98.1:43825)
0924 05:07:40.614057 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.614062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.614073 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:07:40.614127 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.614130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.614146 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.614230 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.614412 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.615531 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.616783 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.617532 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.617605 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.617684 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.618055 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.618298 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260924 05:07:40.618649 19354 tablet_service.cc:1467] Processing CreateTablet for tablet 194c22158a8248a68f56ef6d37c5d87d (DEFAULT_TABLE table=test-table [id=0a250c7fd80e4d47a780d99957529ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:40.619016 19354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194c22158a8248a68f56ef6d37c5d87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:40.618953 19092 tablet_service.cc:1467] Processing CreateTablet for tablet 194c22158a8248a68f56ef6d37c5d87d (DEFAULT_TABLE table=test-table [id=0a250c7fd80e4d47a780d99957529ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:40.619299 19092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194c22158a8248a68f56ef6d37c5d87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:40.621476 19434 tablet_bootstrap.cc:492] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Bootstrap starting.
I20260924 05:07:40.621861 19435 tablet_bootstrap.cc:492] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: Bootstrap starting.
I20260924 05:07:40.621869 19223 tablet_service.cc:1467] Processing CreateTablet for tablet 194c22158a8248a68f56ef6d37c5d87d (DEFAULT_TABLE table=test-table [id=0a250c7fd80e4d47a780d99957529ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:40.622237 19223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194c22158a8248a68f56ef6d37c5d87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:40.622460 19435 tablet_bootstrap.cc:654] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:40.622471 19434 tablet_bootstrap.cc:654] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:40.622731 19435 log.cc:824] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:40.622766 19434 log.cc:824] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:40.623373 19434 tablet_bootstrap.cc:492] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: No bootstrap required, opened a new log
I20260924 05:07:40.623433 19434 ts_tablet_manager.cc:1397] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:40.623462 19435 tablet_bootstrap.cc:492] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: No bootstrap required, opened a new log
I20260924 05:07:40.623519 19435 ts_tablet_manager.cc:1397] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:40.625200 19434 raft_consensus.cc:348] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.625381 19434 raft_consensus.cc:374] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:40.625417 19434 raft_consensus.cc:729] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73394ea54aed464ebbd935a8255ff0b7, State: Initialized, Role: FOLLOWER
I20260924 05:07:40.625551 19434 consensus_queue.cc:260] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.625823 19438 tablet_bootstrap.cc:492] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59: Bootstrap starting.
I20260924 05:07:40.625855 19434 ts_tablet_manager.cc:1428] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:40.625986 19420 heartbeater.cc:503] Master 127.0.98.62:35219 was elected leader, sending a full tablet report...
I20260924 05:07:40.625933 19435 raft_consensus.cc:348] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.626081 19435 raft_consensus.cc:374] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:40.626111 19435 raft_consensus.cc:729] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b35a42c61de4435b6e5983d6dda450e, State: Initialized, Role: FOLLOWER
I20260924 05:07:40.626225 19435 consensus_queue.cc:260] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.626466 19435 ts_tablet_manager.cc:1428] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:07:40.626544 19158 heartbeater.cc:503] Master 127.0.98.62:35219 was elected leader, sending a full tablet report...
I20260924 05:07:40.626817 19438 tablet_bootstrap.cc:654] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:40.627137 19438 log.cc:824] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:40.627862 19438 tablet_bootstrap.cc:492] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59: No bootstrap required, opened a new log
I20260924 05:07:40.627918 19438 ts_tablet_manager.cc:1397] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:07:40.629635 19438 raft_consensus.cc:348] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.629832 19438 raft_consensus.cc:374] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:40.629879 19438 raft_consensus.cc:729] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529ebaf649ee4a10bbd088b388bd3e59, State: Initialized, Role: FOLLOWER
I20260924 05:07:40.630013 19438 consensus_queue.cc:260] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.630256 19438 ts_tablet_manager.cc:1428] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 05:07:40.630311 19289 heartbeater.cc:503] Master 127.0.98.62:35219 was elected leader, sending a full tablet report...
I20260924 05:07:40.652494 19442 raft_consensus.cc:482] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:40.652617 19442 raft_consensus.cc:504] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.652961 19442 leader_election.cc:288] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b35a42c61de4435b6e5983d6dda450e (127.0.98.1:43825), 73394ea54aed464ebbd935a8255ff0b7 (127.0.98.3:43677)
I20260924 05:07:40.655696 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.653110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43677 (local address 127.0.98.2:42861)
0924 05:07:40.653151 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.653163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.653200 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.653380 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.653382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.653383 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.653447 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.653448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.654167 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.654168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.654929 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.654932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.655467 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.655468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.655533 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.655536 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.655582 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.655620 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:07:40.655711 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.653195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42861 (local address 127.0.98.3:43677)
0924 05:07:40.653259 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.653261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.653270 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:40.653294 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.653294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.653301 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.653350 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.653519 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.654090 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.655025 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.655390 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.655438 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.655447 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.655604 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.655640 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:07:40.655750 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.653208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54321 (local address 127.0.98.1:43825)
0924 05:07:40.653261 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.653264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.653497 (+ 233us) server_negotiation.cc:407] Connection header received
0924 05:07:40.653531 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.653532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.653542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.653583 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:40.653739 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.654208 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.655088 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.655431 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.655496 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.655504 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.655654 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.655692 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:07:40.655964 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.653110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43825 (local address 127.0.98.2:54321)
0924 05:07:40.653429 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.653438 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:40.653484 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:40.653603 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:40.653604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:40.653605 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:40.653666 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.653667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.654284 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.654285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.655004 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:40.655007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.655519 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.655520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:40.655579 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.655583 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:40.655635 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:07:40.655665 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":132,"thread_start_us":114,"threads_started":1}
I20260924 05:07:40.656049 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c22158a8248a68f56ef6d37c5d87d" candidate_uuid: "529ebaf649ee4a10bbd088b388bd3e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73394ea54aed464ebbd935a8255ff0b7" is_pre_election: true
I20260924 05:07:40.656208 19374 raft_consensus.cc:2454] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529ebaf649ee4a10bbd088b388bd3e59 in term 0.
I20260924 05:07:40.656299 19112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c22158a8248a68f56ef6d37c5d87d" candidate_uuid: "529ebaf649ee4a10bbd088b388bd3e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b35a42c61de4435b6e5983d6dda450e" is_pre_election: true
I20260924 05:07:40.656406 19177 leader_election.cc:302] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529ebaf649ee4a10bbd088b388bd3e59, 73394ea54aed464ebbd935a8255ff0b7; no voters:
I20260924 05:07:40.656435 19112 raft_consensus.cc:2454] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529ebaf649ee4a10bbd088b388bd3e59 in term 0.
I20260924 05:07:40.656536 19442 raft_consensus.cc:2781] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:07:40.656592 19442 raft_consensus.cc:482] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:07:40.656625 19442 raft_consensus.cc:3037] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.657269 19442 raft_consensus.cc:504] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.657382 19442 leader_election.cc:288] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [CANDIDATE]: Term 1 election: Requested vote from peers 4b35a42c61de4435b6e5983d6dda450e (127.0.98.1:43825), 73394ea54aed464ebbd935a8255ff0b7 (127.0.98.3:43677)
I20260924 05:07:40.657528 19112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c22158a8248a68f56ef6d37c5d87d" candidate_uuid: "529ebaf649ee4a10bbd088b388bd3e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b35a42c61de4435b6e5983d6dda450e"
I20260924 05:07:40.657593 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c22158a8248a68f56ef6d37c5d87d" candidate_uuid: "529ebaf649ee4a10bbd088b388bd3e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73394ea54aed464ebbd935a8255ff0b7"
I20260924 05:07:40.657603 19112 raft_consensus.cc:3037] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.657657 19374 raft_consensus.cc:3037] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:40.658456 19112 raft_consensus.cc:2454] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529ebaf649ee4a10bbd088b388bd3e59 in term 1.
I20260924 05:07:40.658456 19374 raft_consensus.cc:2454] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529ebaf649ee4a10bbd088b388bd3e59 in term 1.
I20260924 05:07:40.658624 19176 leader_election.cc:302] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b35a42c61de4435b6e5983d6dda450e, 529ebaf649ee4a10bbd088b388bd3e59; no voters:
I20260924 05:07:40.658716 19442 raft_consensus.cc:2781] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:40.658880 19442 raft_consensus.cc:686] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [term 1 LEADER]: Becoming Leader. State: Replica: 529ebaf649ee4a10bbd088b388bd3e59, State: Running, Role: LEADER
I20260924 05:07:40.658962 19442 consensus_queue.cc:237] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } }
I20260924 05:07:40.659752 18970 catalog_manager.cc:5495] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 529ebaf649ee4a10bbd088b388bd3e59 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "529ebaf649ee4a10bbd088b388bd3e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ebaf649ee4a10bbd088b388bd3e59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 } health_report { overall_health: UNKNOWN } } }
I20260924 05:07:40.674834 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.672800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.98.2:39447)
0924 05:07:40.672859 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:40.672860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:40.672867 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:07:40.672895 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:40.672896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:40.672911 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:40.672965 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:07:40.673053 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.673490 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.674214 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:40.674391 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:40.674440 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:40.674544 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:07:40.674628 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:07:40.674715 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:40.674755 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 05:07:40.674763 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:07:40.676409 19374 raft_consensus.cc:1261] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 1 FOLLOWER]: Refusing update from remote peer 529ebaf649ee4a10bbd088b388bd3e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:40.676414 19112 raft_consensus.cc:1261] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 1 FOLLOWER]: Refusing update from remote peer 529ebaf649ee4a10bbd088b388bd3e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:40.676733 19444 consensus_queue.cc:1035] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73394ea54aed464ebbd935a8255ff0b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:40.676852 19442 consensus_queue.cc:1035] T 194c22158a8248a68f56ef6d37c5d87d P 529ebaf649ee4a10bbd088b388bd3e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b35a42c61de4435b6e5983d6dda450e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:40.679543 19448 mvcc.cc:205] Tried to move back new op lower bound from 7332767582928359424 to 7332767582859718656. Current Snapshot: MvccSnapshot[applied={T|T < 7332767582928359424}]
I20260924 05:07:40.679656 19449 mvcc.cc:205] Tried to move back new op lower bound from 7332767582928359424 to 7332767582859718656. Current Snapshot: MvccSnapshot[applied={T|T < 7332767582928359424}]
I20260924 05:07:40.681056 18970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:40.681149 18970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:40.681860 18970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dcd9461fa144cc0b98d39efc714e870: Sending DeleteTablet for 3 replicas of tablet 194c22158a8248a68f56ef6d37c5d87d
I20260924 05:07:40.682192 19354 tablet_service.cc:1514] Processing DeleteTablet for tablet 194c22158a8248a68f56ef6d37c5d87d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:40 UTC) from {username='slave'} at 127.0.0.1:53972
I20260924 05:07:40.682197 19092 tablet_service.cc:1514] Processing DeleteTablet for tablet 194c22158a8248a68f56ef6d37c5d87d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:40 UTC) from {username='slave'} at 127.0.0.1:59026
I20260924 05:07:40.682405 19457 tablet_replica.cc:330] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: stopping tablet replica
I20260924 05:07:40.682495 19457 raft_consensus.cc:2229] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:40.682592 19457 raft_consensus.cc:2258] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:40.682641 19223 tablet_service.cc:1514] Processing DeleteTablet for tablet 194c22158a8248a68f56ef6d37c5d87d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:40 UTC) from {username='slave'} at 127.0.0.1:52758
I20260924 05:07:40.682837 19456 tablet_replica.cc:330] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: stopping tablet replica
I20260924 05:07:40.682871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19030
I20260924 05:07:40.682927 19456 raft_consensus.cc:2229] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:40.682972 19457 ts_tablet_manager.cc:1905] T 194c22158a8248a68f56ef6d37c5d87d P 4b35a42c61de4435b6e5983d6dda450e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:40.683009 19456 raft_consensus.cc:2258] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:40.683351 19456 ts_tablet_manager.cc:1905] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:40.684906 19456 ts_tablet_manager.cc:1918] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:40.684967 19456 log.cc:1196] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/194c22158a8248a68f56ef6d37c5d87d
I20260924 05:07:40.685158 19456 ts_tablet_manager.cc:1939] T 194c22158a8248a68f56ef6d37c5d87d P 73394ea54aed464ebbd935a8255ff0b7: Deleting consensus metadata
I20260924 05:07:40.685508 18955 catalog_manager.cc:4841] TS 73394ea54aed464ebbd935a8255ff0b7 (127.0.98.3:43677): tablet 194c22158a8248a68f56ef6d37c5d87d (table test-table [id=0a250c7fd80e4d47a780d99957529ac6]) successfully deleted
W20260924 05:07:40.687582 18954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43825 (error 108)
I20260924 05:07:40.688046 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.687915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43825 (local address 127.0.0.1:59040)
0924 05:07:40.687973 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.688014 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:07:40.688063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19161
W20260924 05:07:40.688747 18954 catalog_manager.cc:4584] TS 4b35a42c61de4435b6e5983d6dda450e (127.0.98.1:43825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 194c22158a8248a68f56ef6d37c5d87d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43825: connect: Connection refused (error 111)
I20260924 05:07:40.692992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19292
I20260924 05:07:40.693012 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:40.692924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39447 (local address 127.0.0.1:52780)
0924 05:07:40.692962 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:40.692988 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:07:40.696374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 18939
2026-09-24T05:07:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (648 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:40 dist-test-slave-m01t krb5kdc[19462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:40 dist-test-slave-m01t krb5kdc[19462](info): set up 2 sockets
Sep 24 05:07:40 dist-test-slave-m01t krb5kdc[19462](info): commencing operation
krb5kdc: starting...
W20260924 05:07:41.641325 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:41 dist-test-slave-m01t krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:07:41.752292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:41.830049 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:41.830261 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:41.830283 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:41.831414 19475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:41.831449 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:41.831461 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:41.831473 19475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:41.831483 19475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:41.832913 19475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:41.833103 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:41.833284 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:41.837930 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:41.837924 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:41.837924 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:41.838016 19475 server_base.cc:1034] running on GCE node
I20260924 05:07:41.838379 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:41.838653 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:41.839792 19475 hybrid_clock.cc:648] HybridClock initialized: now 1790226461839776 us; error 29 us; skew 500 ppm
Sep 24 05:07:41 dist-test-slave-m01t krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:41.842209 19475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:07:41.842608 19475 webserver.cc:460] Webserver started at http://127.0.98.62:37441/ using document root <none> and password file <none>
I20260924 05:07:41.842823 19475 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:41.842869 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:41.842995 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:41.844038 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "443fb51bb9f44488beaa7a5554554dd8"
format_stamp: "Formatted at 2026-09-24 05:07:41 on dist-test-slave-m01t"
server_key: "992356fdc1cd8950b7020de53b3710aa"
server_key_iv: "769432851ffc4abdcb5e922ca33d9b73"
server_key_version: "encryptionkey@0"
I20260924 05:07:41.844373 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "443fb51bb9f44488beaa7a5554554dd8"
format_stamp: "Formatted at 2026-09-24 05:07:41 on dist-test-slave-m01t"
server_key: "992356fdc1cd8950b7020de53b3710aa"
server_key_iv: "769432851ffc4abdcb5e922ca33d9b73"
server_key_version: "encryptionkey@0"
I20260924 05:07:41.845542 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:07:41.846262 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:41.846422 19475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:41.846503 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "443fb51bb9f44488beaa7a5554554dd8"
format_stamp: "Formatted at 2026-09-24 05:07:41 on dist-test-slave-m01t"
server_key: "992356fdc1cd8950b7020de53b3710aa"
server_key_iv: "769432851ffc4abdcb5e922ca33d9b73"
server_key_version: "encryptionkey@0"
I20260924 05:07:41.846568 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:41.868006 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:41.870108 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:41.870214 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:41.873653 19475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38211
I20260924 05:07:41.873697 19542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38211 every 8 connection(s)
I20260924 05:07:41.874034 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:41.874794 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:41.876647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19475
I20260924 05:07:41.876776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:41.876984 392 external_mini_cluster.cc:1428] Setting key b3097cd7ebe7a37a9d2827cf111d3a80
I20260924 05:07:41.877116 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8: Bootstrap starting.
I20260924 05:07:41.877831 19543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:41.878095 19543 log.cc:824] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:41.878741 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8: No bootstrap required, opened a new log
I20260924 05:07:41.880452 19543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443fb51bb9f44488beaa7a5554554dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38211 } }
I20260924 05:07:41.880595 19543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:41.880630 19543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443fb51bb9f44488beaa7a5554554dd8, State: Initialized, Role: FOLLOWER
I20260924 05:07:41.880753 19543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443fb51bb9f44488beaa7a5554554dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38211 } }
I20260924 05:07:41.880829 19543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:41.880860 19543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:41.880904 19543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:41.881428 19543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443fb51bb9f44488beaa7a5554554dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38211 } }
I20260924 05:07:41.881533 19543 leader_election.cc:302] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443fb51bb9f44488beaa7a5554554dd8; no voters:
I20260924 05:07:41.881762 19543 leader_election.cc:288] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:41.881803 19548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:41.881934 19548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [term 1 LEADER]: Becoming Leader. State: Replica: 443fb51bb9f44488beaa7a5554554dd8, State: Running, Role: LEADER
I20260924 05:07:41.882011 19543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:41.882138 19548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443fb51bb9f44488beaa7a5554554dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38211 } }
I20260924 05:07:41.882547 19548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443fb51bb9f44488beaa7a5554554dd8. Latest consensus state: current_term: 1 leader_uuid: "443fb51bb9f44488beaa7a5554554dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443fb51bb9f44488beaa7a5554554dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38211 } } }
I20260924 05:07:41.882553 19550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443fb51bb9f44488beaa7a5554554dd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443fb51bb9f44488beaa7a5554554dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38211 } } }
I20260924 05:07:41.882719 19550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:41.882930 19557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 05:07:41 dist-test-slave-m01t krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:41.883424 19557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:41.884094 19548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:41.885618 19557 catalog_manager.cc:1347] Generated new cluster ID: 7432b21987a74399bad4096b9178754e
I20260924 05:07:41.885665 19557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:41.887285 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:41.877442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.0.98.62:38211)
0924 05:07:41.877612 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:41.877621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:41.877648 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:07:41.878154 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:41.878163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:41.878225 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:41.878339 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:41.878955 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:41.879822 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:41.880558 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:41.881080 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:41.881175 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:41.883828 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:41.883833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:41.883833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:41.883844 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:41.885649 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:41.886058 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:41.886062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:41.886062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:41.886177 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:41.886378 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:41.886379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:41.886379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:41.886700 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:41.886863 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:41.886932 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 05:07:41.887074 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 05:07:41.898988 19557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:41.899482 19557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:41.904130 19557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8: Generated new TSK 0
I20260924 05:07:41.904285 19557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:07:41.932837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38211
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:42.010072 19569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:42.010267 19569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:42.010288 19569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:42.011336 19569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:42.011366 19569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:42.011390 19569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:42.012758 19569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:42.012924 19569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:42.013088 19569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:42.017657 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:42.017655 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.017800 19569 server_base.cc:1034] running on GCE node
W20260924 05:07:42.017652 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.018090 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:42.018321 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:42.019459 19569 hybrid_clock.cc:648] HybridClock initialized: now 1790226462019446 us; error 25 us; skew 500 ppm
Sep 24 05:07:42 dist-test-slave-m01t krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:42.021826 19569 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:07:42.022271 19569 webserver.cc:460] Webserver started at http://127.0.98.1:40791/ using document root <none> and password file <none>
I20260924 05:07:42.022490 19569 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:42.022535 19569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:42.022660 19569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:42.023672 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "1cd54fa779bd4232a43d4707d39c58de"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "003e336721584384137a3260b893de6b"
server_key_iv: "a40526da65668f64fb1c8a73632f08f2"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.024009 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1cd54fa779bd4232a43d4707d39c58de"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "003e336721584384137a3260b893de6b"
server_key_iv: "a40526da65668f64fb1c8a73632f08f2"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.025239 19569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:42.025923 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.026073 19569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:42.026146 19569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "1cd54fa779bd4232a43d4707d39c58de"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "003e336721584384137a3260b893de6b"
server_key_iv: "a40526da65668f64fb1c8a73632f08f2"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.026222 19569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:42.043009 19569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:42.044720 19569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:42.044854 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:42.045086 19569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:42.045389 19569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:42.045423 19569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.045452 19569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:42.045469 19569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.051061 19569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33831
I20260924 05:07:42.051122 19697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33831 every 8 connection(s)
I20260924 05:07:42.051402 19569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:07:42 dist-test-slave-m01t krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:42.057428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19569
I20260924 05:07:42.057542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:07:42.057763 392 external_mini_cluster.cc:1428] Setting key 2a14194d0b7269ae3950184a92b9f441
I20260924 05:07:42.060490 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.052033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48687 (local address 127.0.98.62:38211)
0924 05:07:42.052095 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.052097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.052269 (+ 172us) server_negotiation.cc:407] Connection header received
0924 05:07:42.053192 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.053193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.053205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.053255 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:42.053906 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.054402 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.055398 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.055621 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.055677 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.057764 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:42.057765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:42.057765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:42.057773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:42.059220 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.059569 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.059570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.059571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.059637 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.059864 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.059865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.059866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.059980 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:42.060038 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.060301 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.060397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:07:42.060611 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.052005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38211 (local address 127.0.98.1:48687)
0924 05:07:42.052147 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.052188 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.053068 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.053324 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.053329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.053580 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:42.053802 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.053807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.054491 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.054492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.055306 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.055308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.055697 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.055698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.055814 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.056268 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:42.056276 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:42.057652 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:42.059296 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:42.059302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:42.059303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:42.059502 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:42.059699 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:42.059699 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:42.059699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:42.059787 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:42.060042 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:42.060049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:42.060180 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.060265 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.060406 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260924 05:07:42.061031 19698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38211
I20260924 05:07:42.061124 19698 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:42.061298 19698 heartbeater.cc:511] Master 127.0.98.62:38211 requested a full tablet report, sending...
I20260924 05:07:42.061787 19507 ts_manager.cc:194] Registered new tserver with Master: 1cd54fa779bd4232a43d4707d39c58de (127.0.98.1:33831)
I20260924 05:07:42.062382 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48687
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:07:42.082003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38211
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:42.162123 19703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:42.162318 19703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:42.162340 19703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:42.163461 19703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:42.163502 19703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:42.163530 19703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:42.164976 19703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:42.165194 19703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:42.165390 19703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:42.169993 19708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:42.169978 19711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.170125 19703 server_base.cc:1034] running on GCE node
W20260924 05:07:42.170182 19709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.170363 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:42.170589 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:42.171728 19703 hybrid_clock.cc:648] HybridClock initialized: now 1790226462171718 us; error 27 us; skew 500 ppm
Sep 24 05:07:42 dist-test-slave-m01t krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:42.174001 19703 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:07:42.174401 19703 webserver.cc:460] Webserver started at http://127.0.98.2:45929/ using document root <none> and password file <none>
I20260924 05:07:42.174575 19703 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:42.174615 19703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:42.174702 19703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:42.175639 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c45b82ac66142eebe64a6f0395e38b7"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "9f6955c96aada45026e536bceeddd667"
server_key_iv: "4a5cdaf566c41a275e62ba11afdd17ae"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.175916 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c45b82ac66142eebe64a6f0395e38b7"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "9f6955c96aada45026e536bceeddd667"
server_key_iv: "4a5cdaf566c41a275e62ba11afdd17ae"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.177078 19703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:42.177685 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.177836 19703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.177915 19703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "9c45b82ac66142eebe64a6f0395e38b7"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "9f6955c96aada45026e536bceeddd667"
server_key_iv: "4a5cdaf566c41a275e62ba11afdd17ae"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.177975 19703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:42.194648 19703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:42.196331 19703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:42.196472 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:42.196696 19703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:42.197026 19703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:42.197072 19703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.197117 19703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:42.197139 19703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.202714 19703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45213
I20260924 05:07:42.202778 19831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45213 every 8 connection(s)
I20260924 05:07:42.203053 19703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:42.206503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19703
I20260924 05:07:42.206622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:07:42.206799 392 external_mini_cluster.cc:1428] Setting key b5437fe340878e7a0ccf1c96c4f7fc4d
Sep 24 05:07:42 dist-test-slave-m01t krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:42.211980 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.203765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40337 (local address 127.0.98.62:38211)
0924 05:07:42.203853 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.203855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.204009 (+ 154us) server_negotiation.cc:407] Connection header received
0924 05:07:42.204913 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.204914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.204927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.204973 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:42.205571 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.206113 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.207225 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.207484 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.207538 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.209491 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:42.209492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:42.209492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:42.209500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:42.210780 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.211135 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.211137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.211137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.211191 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.211401 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.211401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.211402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.211509 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:42.211564 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.211818 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.211895 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 05:07:42.212131 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.203699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38211 (local address 127.0.98.2:40337)
0924 05:07:42.203892 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.203929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.204807 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.205054 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.205058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.205298 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:42.205480 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.205484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.206199 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.206200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.207153 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.207155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.207556 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.207557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.207689 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.208151 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:42.208159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:42.209403 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:42.210856 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:42.210863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:42.210865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:42.211067 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:42.211247 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:42.211248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:42.211248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:42.211341 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:42.211566 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:42.211572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:42.211700 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.211784 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.211927 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260924 05:07:42.212944 19832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38211
I20260924 05:07:42.213021 19832 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:42.213182 19832 heartbeater.cc:511] Master 127.0.98.62:38211 requested a full tablet report, sending...
I20260924 05:07:42.213543 19507 ts_manager.cc:194] Registered new tserver with Master: 9c45b82ac66142eebe64a6f0395e38b7 (127.0.98.2:45213)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:07:42.214054 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:07:42.228626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38211
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:42.307024 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:42.307207 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:42.307228 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:42.308286 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:42.308317 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:42.308341 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:42.309744 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:42.309942 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:42.310115 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:42.314674 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.314757 19837 server_base.cc:1034] running on GCE node
W20260924 05:07:42.314697 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:42.314697 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.315074 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:42.315307 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:42.316443 19837 hybrid_clock.cc:648] HybridClock initialized: now 1790226462316416 us; error 41 us; skew 500 ppm
Sep 24 05:07:42 dist-test-slave-m01t krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:42.318913 19837 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:07:42.319360 19837 webserver.cc:460] Webserver started at http://127.0.98.3:42159/ using document root <none> and password file <none>
I20260924 05:07:42.319573 19837 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:42.319623 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:42.319756 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:42.320791 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c31592cc2964e8283193231a10d17e1"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "92742d4195d4b7f3e3a9009145d46e4c"
server_key_iv: "f289409560c06ff638251d2da6af7a6c"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.321111 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c31592cc2964e8283193231a10d17e1"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "92742d4195d4b7f3e3a9009145d46e4c"
server_key_iv: "f289409560c06ff638251d2da6af7a6c"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.322422 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:07:42.323117 19852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.323277 19837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:42.323359 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "9c31592cc2964e8283193231a10d17e1"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "92742d4195d4b7f3e3a9009145d46e4c"
server_key_iv: "f289409560c06ff638251d2da6af7a6c"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.323421 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:42.338053 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:42.339813 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:42.339950 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:42.340163 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:42.340457 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:42.340490 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.340518 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:42.340535 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.345664 19837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42439
I20260924 05:07:42.345755 19965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42439 every 8 connection(s)
I20260924 05:07:42.346016 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:07:42 dist-test-slave-m01t krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:42.352954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 19837
I20260924 05:07:42.353071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:07:42.353269 392 external_mini_cluster.cc:1428] Setting key b85e076bbffe9dd9c9832abb6ffe4466
I20260924 05:07:42.354714 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.346659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58457 (local address 127.0.98.62:38211)
0924 05:07:42.346724 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.346726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.346895 (+ 169us) server_negotiation.cc:407] Connection header received
0924 05:07:42.347760 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.347761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.347769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.347820 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:42.348388 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.348935 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.349996 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.350177 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.350227 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.352107 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:42.352108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:42.352108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:42.352113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:42.353347 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.353809 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.353810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.353811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.353865 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.354104 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.354105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.354106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.354230 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:42.354291 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.354547 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.354618 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:07:42.354861 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.346618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38211 (local address 127.0.98.3:58457)
0924 05:07:42.346773 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.346815 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.347680 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.347876 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.347879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.348129 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:42.348313 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.348317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.349001 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.349002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.349933 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.349936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.350237 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.350238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.350332 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.350789 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:07:42.350797 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:42.352035 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:42.353433 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:42.353440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:42.353442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:42.353758 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:42.353933 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:07:42.353934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:07:42.353934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:07:42.354040 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:07:42.354289 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:42.354296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:42.354426 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.354512 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.354656 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260924 05:07:42.355242 19966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38211
I20260924 05:07:42.355329 19966 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:42.355491 19966 heartbeater.cc:511] Master 127.0.98.62:38211 requested a full tablet report, sending...
I20260924 05:07:42.355835 19507 ts_manager.cc:194] Registered new tserver with Master: 9c31592cc2964e8283193231a10d17e1 (127.0.98.3:42439)
I20260924 05:07:42.356180 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58457
I20260924 05:07:42.365899 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:07:42.372255 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.367181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.0.98.62:38211)
0924 05:07:42.367232 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.367234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.367284 (+ 50us) server_negotiation.cc:407] Connection header received
0924 05:07:42.367328 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.367329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.367337 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.367386 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:42.367868 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.368316 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.369049 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.369226 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.369273 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.370382 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:42.370383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:42.370383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:42.370389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:42.371419 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.371709 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.371710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.371710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.371751 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:42.371881 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:42.371882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:42.371882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:42.371997 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:42.372065 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.372137 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.372191 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:07:42.373566 19507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:07:42.375128 19507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:07:42.381016 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.377039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42439 (local address 127.0.0.1:60498)
0924 05:07:42.377484 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.377493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.377530 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.377768 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.377769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.377771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:42.377808 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.377809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.378709 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.378710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.379838 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.379840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.380733 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.380734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.380815 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.380829 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.380899 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.380945 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":78,"threads_started":1}
I20260924 05:07:42.381219 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.377034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45213 (local address 127.0.0.1:54024)
0924 05:07:42.377279 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.377317 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.377379 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.377696 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.377697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.377699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:42.377772 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.377774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.379119 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.379120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.380085 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.380087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.381001 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.381002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.381067 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.381071 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.381121 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.381157 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":61,"threads_started":1}
I20260924 05:07:42.381258 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.377181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.0.98.3:42439)
0924 05:07:42.377504 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.377510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.377522 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:07:42.377593 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.377595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.377608 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.377673 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:42.377859 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.378605 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.379944 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.380651 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.380722 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.380780 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.380929 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.380968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":78,"threads_started":1}
I20260924 05:07:42.381441 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.377200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.98.2:45213)
0924 05:07:42.377546 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.377549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.377558 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:42.377591 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.377593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.377606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.377847 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:42.377928 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.379012 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.380158 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.380858 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.380913 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.380967 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.381324 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.381371 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":74,"threads_started":1}
I20260924 05:07:42.381928 19900 tablet_service.cc:1467] Processing CreateTablet for tablet da02fd5a86bf4d5683e2f18a9ace22de (DEFAULT_TABLE table=test-table [id=ce92b16f26b9426ea8d6565007d09e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:42.382128 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.377034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33831 (local address 127.0.0.1:39270)
0924 05:07:42.377354 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.377364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.377415 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.377660 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.377663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.377665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:42.377757 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.377759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.380147 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.380148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.380958 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.380960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.381892 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.381893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.381978 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.381982 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.382027 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.382064 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260924 05:07:42.382151 19766 tablet_service.cc:1467] Processing CreateTablet for tablet da02fd5a86bf4d5683e2f18a9ace22de (DEFAULT_TABLE table=test-table [id=ce92b16f26b9426ea8d6565007d09e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:42.382279 19900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da02fd5a86bf4d5683e2f18a9ace22de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:42.382405 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.377280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.0.98.1:33831)
0924 05:07:42.377485 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.377491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.377509 (+ 18us) server_negotiation.cc:407] Connection header received
0924 05:07:42.377547 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.377549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.377565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.377652 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:42.377933 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.379016 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.381038 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.381751 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.381808 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.381862 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.382257 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.382331 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260924 05:07:42.382426 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da02fd5a86bf4d5683e2f18a9ace22de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:42.382959 19632 tablet_service.cc:1467] Processing CreateTablet for tablet da02fd5a86bf4d5683e2f18a9ace22de (DEFAULT_TABLE table=test-table [id=ce92b16f26b9426ea8d6565007d09e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:42.383301 19632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da02fd5a86bf4d5683e2f18a9ace22de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:42.385495 19981 tablet_bootstrap.cc:492] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Bootstrap starting.
I20260924 05:07:42.385908 19982 tablet_bootstrap.cc:492] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: Bootstrap starting.
I20260924 05:07:42.386228 19981 tablet_bootstrap.cc:654] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:42.386431 19982 tablet_bootstrap.cc:654] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:42.386714 19981 log.cc:824] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:42.386914 19982 log.cc:824] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:42.387001 19980 tablet_bootstrap.cc:492] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Bootstrap starting.
I20260924 05:07:42.387475 19981 tablet_bootstrap.cc:492] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: No bootstrap required, opened a new log
I20260924 05:07:42.387527 19981 ts_tablet_manager.cc:1397] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:42.387540 19982 tablet_bootstrap.cc:492] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: No bootstrap required, opened a new log
I20260924 05:07:42.387602 19982 ts_tablet_manager.cc:1397] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:42.388950 19981 raft_consensus.cc:348] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.389092 19981 raft_consensus.cc:374] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:42.389114 19981 raft_consensus.cc:729] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c31592cc2964e8283193231a10d17e1, State: Initialized, Role: FOLLOWER
I20260924 05:07:42.389065 19982 raft_consensus.cc:348] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.389209 19981 consensus_queue.cc:260] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.389233 19982 raft_consensus.cc:374] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:42.389277 19982 raft_consensus.cc:729] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd54fa779bd4232a43d4707d39c58de, State: Initialized, Role: FOLLOWER
I20260924 05:07:42.389380 19982 consensus_queue.cc:260] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.389454 19981 ts_tablet_manager.cc:1428] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:42.389459 19966 heartbeater.cc:503] Master 127.0.98.62:38211 was elected leader, sending a full tablet report...
I20260924 05:07:42.389639 19982 ts_tablet_manager.cc:1428] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:42.389787 19698 heartbeater.cc:503] Master 127.0.98.62:38211 was elected leader, sending a full tablet report...
I20260924 05:07:42.390132 19980 tablet_bootstrap.cc:654] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:42.390621 19980 log.cc:824] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:42.391304 19980 tablet_bootstrap.cc:492] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: No bootstrap required, opened a new log
I20260924 05:07:42.391371 19980 ts_tablet_manager.cc:1397] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 05:07:42.392750 19980 raft_consensus.cc:348] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.392902 19980 raft_consensus.cc:374] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:42.392936 19980 raft_consensus.cc:729] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c45b82ac66142eebe64a6f0395e38b7, State: Initialized, Role: FOLLOWER
I20260924 05:07:42.393038 19980 consensus_queue.cc:260] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.393270 19980 ts_tablet_manager.cc:1428] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:07:42.393306 19832 heartbeater.cc:503] Master 127.0.98.62:38211 was elected leader, sending a full tablet report...
W20260924 05:07:42.453581 19833 tablet.cc:2369] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:07:42.552060 19699 tablet.cc:2369] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:07:42.596537 19967 tablet.cc:2369] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:07:42.666561 19985 raft_consensus.cc:482] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:42.666683 19985 raft_consensus.cc:504] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.667037 19985 leader_election.cc:288] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c45b82ac66142eebe64a6f0395e38b7 (127.0.98.2:45213), 1cd54fa779bd4232a43d4707d39c58de (127.0.98.1:33831)
I20260924 05:07:42.669903 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.667376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56617 (local address 127.0.98.2:45213)
0924 05:07:42.667430 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.667431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.667439 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:42.667461 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.667462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.667488 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.667554 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:42.667718 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.668185 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.669151 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.669565 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.669624 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.669635 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.669809 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.669844 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:07:42.669903 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.667259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45213 (local address 127.0.98.3:56617)
0924 05:07:42.667304 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.667315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.667358 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.667573 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.667574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.667577 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:42.667646 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.667648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.668258 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.668259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.669066 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.669069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.669640 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.669641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.669747 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.669751 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.669794 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.669829 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 05:07:42.670008 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.667304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48185 (local address 127.0.98.1:33831)
0924 05:07:42.667346 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.667348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.667648 (+ 300us) server_negotiation.cc:407] Connection header received
0924 05:07:42.667681 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.667682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.667693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.667738 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:42.667880 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.668333 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.669248 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.669673 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.669761 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.669770 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.669916 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.669949 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:07:42.670013 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.667259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33831 (local address 127.0.98.3:48185)
0924 05:07:42.667567 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.667577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:42.667630 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:42.667753 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:42.667754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:42.667755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:42.667819 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.667821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.668414 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.668415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.669159 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:42.669161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.669768 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.669769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:42.669849 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.669852 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:42.669897 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:07:42.669934 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":143,"thread_start_us":123,"threads_started":1}
I20260924 05:07:42.670225 19786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02fd5a86bf4d5683e2f18a9ace22de" candidate_uuid: "9c31592cc2964e8283193231a10d17e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b82ac66142eebe64a6f0395e38b7" is_pre_election: true
I20260924 05:07:42.670372 19786 raft_consensus.cc:2454] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c31592cc2964e8283193231a10d17e1 in term 0.
I20260924 05:07:42.670363 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02fd5a86bf4d5683e2f18a9ace22de" candidate_uuid: "9c31592cc2964e8283193231a10d17e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd54fa779bd4232a43d4707d39c58de" is_pre_election: true
I20260924 05:07:42.670502 19652 raft_consensus.cc:2454] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c31592cc2964e8283193231a10d17e1 in term 0.
I20260924 05:07:42.670608 19854 leader_election.cc:302] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c31592cc2964e8283193231a10d17e1, 9c45b82ac66142eebe64a6f0395e38b7; no voters:
I20260924 05:07:42.670763 19985 raft_consensus.cc:2781] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:07:42.670802 19985 raft_consensus.cc:482] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:07:42.670816 19985 raft_consensus.cc:3037] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:42.671471 19985 raft_consensus.cc:504] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.671581 19985 leader_election.cc:288] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9c45b82ac66142eebe64a6f0395e38b7 (127.0.98.2:45213), 1cd54fa779bd4232a43d4707d39c58de (127.0.98.1:33831)
I20260924 05:07:42.671726 19786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02fd5a86bf4d5683e2f18a9ace22de" candidate_uuid: "9c31592cc2964e8283193231a10d17e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c45b82ac66142eebe64a6f0395e38b7"
I20260924 05:07:42.671773 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da02fd5a86bf4d5683e2f18a9ace22de" candidate_uuid: "9c31592cc2964e8283193231a10d17e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd54fa779bd4232a43d4707d39c58de"
I20260924 05:07:42.671795 19786 raft_consensus.cc:3037] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:42.671835 19652 raft_consensus.cc:3037] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:42.672452 19786 raft_consensus.cc:2454] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c31592cc2964e8283193231a10d17e1 in term 1.
I20260924 05:07:42.672472 19652 raft_consensus.cc:2454] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c31592cc2964e8283193231a10d17e1 in term 1.
I20260924 05:07:42.672612 19853 leader_election.cc:302] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd54fa779bd4232a43d4707d39c58de, 9c31592cc2964e8283193231a10d17e1; no voters:
I20260924 05:07:42.672701 19985 raft_consensus.cc:2781] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:42.672832 19985 raft_consensus.cc:686] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 1 LEADER]: Becoming Leader. State: Replica: 9c31592cc2964e8283193231a10d17e1, State: Running, Role: LEADER
I20260924 05:07:42.672914 19985 consensus_queue.cc:237] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } }
I20260924 05:07:42.673777 19506 catalog_manager.cc:5495] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c31592cc2964e8283193231a10d17e1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c31592cc2964e8283193231a10d17e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c31592cc2964e8283193231a10d17e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } health_report { overall_health: UNKNOWN } } }
I20260924 05:07:42.731233 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.729096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.98.3:42439)
0924 05:07:42.729154 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.729156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.729164 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:42.729198 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.729199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.729216 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.729275 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:07:42.729373 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.729898 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.730624 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.730805 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.730849 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.730956 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:07:42.731061 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:07:42.731120 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.731134 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.731149 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:07:42.732969 19652 raft_consensus.cc:1261] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 1 FOLLOWER]: Refusing update from remote peer 9c31592cc2964e8283193231a10d17e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:42.732969 19786 raft_consensus.cc:1261] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 1 FOLLOWER]: Refusing update from remote peer 9c31592cc2964e8283193231a10d17e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:42.733287 19990 consensus_queue.cc:1035] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd54fa779bd4232a43d4707d39c58de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:42.733428 19985 consensus_queue.cc:1035] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c45b82ac66142eebe64a6f0395e38b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:42.735782 19994 mvcc.cc:205] Tried to move back new op lower bound from 7332767591351541760 to 7332767591108739072. Current Snapshot: MvccSnapshot[applied={T|T < 7332767591351541760}]
I20260924 05:07:42.735824 19993 mvcc.cc:205] Tried to move back new op lower bound from 7332767591351541760 to 7332767591108739072. Current Snapshot: MvccSnapshot[applied={T|T < 7332767591351541760}]
I20260924 05:07:42.735953 19995 mvcc.cc:205] Tried to move back new op lower bound from 7332767591351541760 to 7332767591108739072. Current Snapshot: MvccSnapshot[applied={T|T < 7332767591351541760}]
I20260924 05:07:42.738152 19506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:42.738260 19506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:42.739001 19506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443fb51bb9f44488beaa7a5554554dd8: Sending DeleteTablet for 3 replicas of tablet da02fd5a86bf4d5683e2f18a9ace22de
I20260924 05:07:42.739305 19766 tablet_service.cc:1514] Processing DeleteTablet for tablet da02fd5a86bf4d5683e2f18a9ace22de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54024
I20260924 05:07:42.739355 19632 tablet_service.cc:1514] Processing DeleteTablet for tablet da02fd5a86bf4d5683e2f18a9ace22de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39270
I20260924 05:07:42.739620 19900 tablet_service.cc:1514] Processing DeleteTablet for tablet da02fd5a86bf4d5683e2f18a9ace22de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60498
I20260924 05:07:42.739688 20004 tablet_replica.cc:330] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de: stopping tablet replica
I20260924 05:07:42.739780 20004 raft_consensus.cc:2229] T da02fd5a86bf4d5683e2f18a9ace22de P 1cd54fa779bd4232a43d4707d39c58de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:42.739892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19569
I20260924 05:07:42.739564 20003 tablet_replica.cc:330] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: stopping tablet replica
I20260924 05:07:42.741066 20003 raft_consensus.cc:2229] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:42.741248 20003 raft_consensus.cc:2258] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:42.741863 20003 ts_tablet_manager.cc:1905] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:42.743409 20003 ts_tablet_manager.cc:1918] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:42.743527 20003 log.cc:1196] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-1/wal/wals/da02fd5a86bf4d5683e2f18a9ace22de
I20260924 05:07:42.743816 20003 ts_tablet_manager.cc:1939] T da02fd5a86bf4d5683e2f18a9ace22de P 9c45b82ac66142eebe64a6f0395e38b7: Deleting consensus metadata
I20260924 05:07:42.741609 20005 tablet_replica.cc:330] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: stopping tablet replica
I20260924 05:07:42.744311 20005 raft_consensus.cc:2229] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:07:42.744290 19492 catalog_manager.cc:4841] TS 9c45b82ac66142eebe64a6f0395e38b7 (127.0.98.2:45213): tablet da02fd5a86bf4d5683e2f18a9ace22de (table test-table [id=ce92b16f26b9426ea8d6565007d09e41]) successfully deleted
I20260924 05:07:42.744482 20005 raft_consensus.cc:2258] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:42.744958 20005 ts_tablet_manager.cc:1905] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:07:42.746332 19491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33831 (error 108)
I20260924 05:07:42.746590 20005 ts_tablet_manager.cc:1918] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:42.746721 20005 log.cc:1196] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/da02fd5a86bf4d5683e2f18a9ace22de
I20260924 05:07:42.746817 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.746689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33831 (local address 127.0.0.1:39276)
0924 05:07:42.746748 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:42.746786 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 05:07:42.746940 19491 catalog_manager.cc:4584] TS 1cd54fa779bd4232a43d4707d39c58de (127.0.98.1:33831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da02fd5a86bf4d5683e2f18a9ace22de: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33831: connect: Connection refused (error 111)
I20260924 05:07:42.747164 20005 ts_tablet_manager.cc:1939] T da02fd5a86bf4d5683e2f18a9ace22de P 9c31592cc2964e8283193231a10d17e1: Deleting consensus metadata
I20260924 05:07:42.747210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19703
I20260924 05:07:42.749928 19493 catalog_manager.cc:4841] TS 9c31592cc2964e8283193231a10d17e1 (127.0.98.3:42439): tablet da02fd5a86bf4d5683e2f18a9ace22de (table test-table [id=ce92b16f26b9426ea8d6565007d09e41]) successfully deleted
I20260924 05:07:42.753013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19837
I20260924 05:07:42.757064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 19475
2026-09-24T05:07:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2061 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T05:07:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:42Z Disabled control of system clock
I20260924 05:07:42.781498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33149
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:42.859915 20010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:42.860145 20010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:42.860165 20010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:42.861260 20010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:42.861294 20010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:42.861304 20010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:42.861315 20010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:42.861326 20010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:42.862622 20010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:42.862807 20010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:42.863035 20010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:42.867453 20016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:42.867445 20015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.867524 20010 server_base.cc:1034] running on GCE node
W20260924 05:07:42.867508 20018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:42.867873 20010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:42.868135 20010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:42.869279 20010 hybrid_clock.cc:648] HybridClock initialized: now 1790226462869268 us; error 27 us; skew 500 ppm
I20260924 05:07:42.870442 20010 webserver.cc:460] Webserver started at http://127.0.98.62:43995/ using document root <none> and password file <none>
I20260924 05:07:42.870644 20010 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:42.870704 20010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:42.870833 20010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:42.871821 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "a044c7f3fa074ab0a06c0da209678c01"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "f3e9ec4026b514ecba0777806b50fecd"
server_key_iv: "199f48d50b3354e12d953d240fdcca8f"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.872134 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "a044c7f3fa074ab0a06c0da209678c01"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "f3e9ec4026b514ecba0777806b50fecd"
server_key_iv: "199f48d50b3354e12d953d240fdcca8f"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.873257 20010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:42.873903 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:42.874025 20010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:42.874097 20010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "a044c7f3fa074ab0a06c0da209678c01"
format_stamp: "Formatted at 2026-09-24 05:07:42 on dist-test-slave-m01t"
server_key: "f3e9ec4026b514ecba0777806b50fecd"
server_key_iv: "199f48d50b3354e12d953d240fdcca8f"
server_key_version: "encryptionkey@0"
I20260924 05:07:42.874161 20010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:42.884660 20010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:42.884907 20010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:42.885025 20010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:42.888569 20010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33149
I20260924 05:07:42.888628 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33149 every 8 connection(s)
I20260924 05:07:42.888896 20010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:42.889835 20077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:42.891978 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01: Bootstrap starting.
I20260924 05:07:42.892513 20077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:42.892776 20077 log.cc:824] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:42.893364 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01: No bootstrap required, opened a new log
I20260924 05:07:42.894930 20077 raft_consensus.cc:348] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044c7f3fa074ab0a06c0da209678c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33149 } }
I20260924 05:07:42.895059 20077 raft_consensus.cc:374] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:42.895081 20077 raft_consensus.cc:729] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a044c7f3fa074ab0a06c0da209678c01, State: Initialized, Role: FOLLOWER
I20260924 05:07:42.895219 20077 consensus_queue.cc:260] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044c7f3fa074ab0a06c0da209678c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33149 } }
I20260924 05:07:42.895280 20077 raft_consensus.cc:388] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:42.895301 20077 raft_consensus.cc:482] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:42.895345 20077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:42.895542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20010
I20260924 05:07:42.895620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:42.895795 392 external_mini_cluster.cc:1428] Setting key d9c3c66a0c9f3ec6902d5daa417ad4e7
I20260924 05:07:42.895869 20077 raft_consensus.cc:504] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044c7f3fa074ab0a06c0da209678c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33149 } }
I20260924 05:07:42.895967 20077 leader_election.cc:302] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a044c7f3fa074ab0a06c0da209678c01; no voters:
I20260924 05:07:42.896126 20077 leader_election.cc:288] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:42.896241 20080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:42.896365 20077 sys_catalog.cc:564] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:42.896379 20080 raft_consensus.cc:686] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [term 1 LEADER]: Becoming Leader. State: Replica: a044c7f3fa074ab0a06c0da209678c01, State: Running, Role: LEADER
I20260924 05:07:42.896461 20080 consensus_queue.cc:237] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044c7f3fa074ab0a06c0da209678c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33149 } }
I20260924 05:07:42.896899 20082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a044c7f3fa074ab0a06c0da209678c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044c7f3fa074ab0a06c0da209678c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33149 } } }
I20260924 05:07:42.896976 20082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:42.897400 20086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:42.897918 20086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:42.898401 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a044c7f3fa074ab0a06c0da209678c01. Latest consensus state: current_term: 1 leader_uuid: "a044c7f3fa074ab0a06c0da209678c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044c7f3fa074ab0a06c0da209678c01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33149 } } }
I20260924 05:07:42.898823 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:42.901014 20086 catalog_manager.cc:1347] Generated new cluster ID: 50b5ba031874452081f6fe236834e33c
I20260924 05:07:42.901068 20086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:42.901528 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:42.896270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.98.62:33149)
0924 05:07:42.896682 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:42.896689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:42.896712 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:07:42.896817 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:42.896825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:42.896900 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:42.897295 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:42.897410 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.898819 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.899597 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:42.900012 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:42.900114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:42.900788 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:42.900791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:42.900791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:42.900978 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:42.901019 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:42.901033 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:42.901259 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:42.901293 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:07:42.901366 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":75,"threads_started":1}
I20260924 05:07:42.927977 20086 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:42.928506 20086 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:42.937183 20086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01: Generated new TSK 0
I20260924 05:07:42.937379 20086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:42.944017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33149
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:07:43.021445 20101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:43.021660 20101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:43.021682 20101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:43.022887 20101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:43.022931 20101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:43.022957 20101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:07:43.024318 20101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:43.024557 20101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:43.024857 20101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:43.029598 20109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:43.029615 20106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:43.029615 20107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:43.029834 20101 server_base.cc:1034] running on GCE node
I20260924 05:07:43.030036 20101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:43.030282 20101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:43.031435 20101 hybrid_clock.cc:648] HybridClock initialized: now 1790226463031411 us; error 33 us; skew 500 ppm
I20260924 05:07:43.032689 20101 webserver.cc:460] Webserver started at http://127.0.98.1:44657/ using document root <none> and password file <none>
I20260924 05:07:43.032919 20101 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:43.032968 20101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:43.033102 20101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:43.034241 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4550cfaff2b4b5db76aee03fffcf2a8"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "a4a2c90e06c8dfc27b9389e38aa76fdc"
server_key_iv: "85c71f46413eee334d6e1f7422976f5c"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.034571 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4550cfaff2b4b5db76aee03fffcf2a8"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "a4a2c90e06c8dfc27b9389e38aa76fdc"
server_key_iv: "85c71f46413eee334d6e1f7422976f5c"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.035778 20101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:43.036414 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.036576 20101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:43.036659 20101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal
uuid: "e4550cfaff2b4b5db76aee03fffcf2a8"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "a4a2c90e06c8dfc27b9389e38aa76fdc"
server_key_iv: "85c71f46413eee334d6e1f7422976f5c"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.036718 20101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:43.049309 20101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:43.049577 20101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:43.049700 20101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:43.049937 20101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:43.050225 20101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:43.050261 20101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.050280 20101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:43.050297 20101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.055490 20101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37743
I20260924 05:07:43.055617 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37743 every 8 connection(s)
I20260924 05:07:43.055804 20101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:07:43.058506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20101
I20260924 05:07:43.058615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:07:43.058811 392 external_mini_cluster.cc:1428] Setting key 8e88e3242ce2f5e851b9a3c9a08d45f6
I20260924 05:07:43.059609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33149
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:07:43.061066 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.056449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58173 (local address 127.0.98.62:33149)
0924 05:07:43.056493 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.056495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.056780 (+ 285us) server_negotiation.cc:407] Connection header received
0924 05:07:43.057279 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.057280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.057295 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.057345 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:43.058057 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.058698 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.059715 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.059974 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.060043 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.060810 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:43.060811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:43.060811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:43.060816 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:43.060830 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:43.060835 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:43.060880 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.060957 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.060988 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:07:43.061275 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.056431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33149 (local address 127.0.98.1:58173)
0924 05:07:43.056622 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.056663 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.057177 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.057412 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.057416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.057426 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:43.057962 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.057967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.058791 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.058793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.059616 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.059620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.060058 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.060059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.060174 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.060672 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:43.060672 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:43.060702 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:43.060724 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:43.060892 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:43.060896 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:43.060899 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.060959 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.061057 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260924 05:07:43.061697 20229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33149
I20260924 05:07:43.061806 20229 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:43.061973 20229 heartbeater.cc:511] Master 127.0.98.62:33149 requested a full tablet report, sending...
I20260924 05:07:43.062441 20041 ts_manager.cc:194] Registered new tserver with Master: e4550cfaff2b4b5db76aee03fffcf2a8 (127.0.98.1:37743)
I20260924 05:07:43.063024 20041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58173
W20260924 05:07:43.138760 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:43.138980 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:43.139009 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:43.140254 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:43.140295 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:43.140328 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:07:43.141655 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:43.141912 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:43.142179 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:43.146836 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:43.146862 20232 server_base.cc:1034] running on GCE node
W20260924 05:07:43.146831 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:43.146827 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:43.147240 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:43.147459 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:43.148591 20232 hybrid_clock.cc:648] HybridClock initialized: now 1790226463148572 us; error 31 us; skew 500 ppm
I20260924 05:07:43.149684 20232 webserver.cc:460] Webserver started at http://127.0.98.2:36305/ using document root <none> and password file <none>
I20260924 05:07:43.149917 20232 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:43.149966 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:43.150094 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:43.151095 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a4ea9f5b0284865b0acda67481f0289"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "306616c011ec6079e809aecf47868197"
server_key_iv: "db111b36cbe6e81ceacffd080aec3709"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.151422 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a4ea9f5b0284865b0acda67481f0289"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "306616c011ec6079e809aecf47868197"
server_key_iv: "db111b36cbe6e81ceacffd080aec3709"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.152552 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:07:43.153173 20246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.153334 20232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:43.153410 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal
uuid: "1a4ea9f5b0284865b0acda67481f0289"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "306616c011ec6079e809aecf47868197"
server_key_iv: "db111b36cbe6e81ceacffd080aec3709"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.153477 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:43.195945 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:43.196260 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:43.196391 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:43.196625 20232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:43.197027 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:43.197065 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.197095 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:43.197119 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.202450 20232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40539
I20260924 05:07:43.202497 20359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40539 every 8 connection(s)
I20260924 05:07:43.202803 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:07:43.204708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20232
I20260924 05:07:43.204825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:07:43.205009 392 external_mini_cluster.cc:1428] Setting key 1a4c3cea3bc64a53c22384e56dacabbd
I20260924 05:07:43.205984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33149
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:07:43.208298 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.203461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35099 (local address 127.0.98.62:33149)
0924 05:07:43.203524 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.203526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.203745 (+ 219us) server_negotiation.cc:407] Connection header received
0924 05:07:43.204217 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.204218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.204227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.204280 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:43.205033 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.205614 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.206959 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.207171 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.207238 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.208030 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:43.208032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:43.208032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:43.208036 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:43.208054 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:43.208058 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:43.208124 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.208200 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.208225 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:07:43.208424 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.203446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33149 (local address 127.0.98.2:35099)
0924 05:07:43.203623 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.203664 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.204131 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.204335 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.204338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.204346 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:43.204945 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.204952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.205703 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.205705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.206878 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.206882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.207264 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.207266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.207402 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.207892 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:43.207893 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:43.207920 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:43.207942 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:43.208125 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:43.208127 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:43.208129 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.208179 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.208251 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260924 05:07:43.208812 20360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33149
I20260924 05:07:43.208889 20360 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:43.209040 20360 heartbeater.cc:511] Master 127.0.98.62:33149 requested a full tablet report, sending...
I20260924 05:07:43.209363 20041 ts_manager.cc:194] Registered new tserver with Master: 1a4ea9f5b0284865b0acda67481f0289 (127.0.98.2:40539)
I20260924 05:07:43.209733 20041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35099
W20260924 05:07:43.285846 20363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:43.286046 20363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:43.286074 20363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:43.287212 20363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:43.287256 20363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:43.287287 20363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:07:43.288628 20363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:43.288874 20363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:43.289158 20363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:43.293795 20371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:43.293848 20369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:43.293946 20363 server_base.cc:1034] running on GCE node
W20260924 05:07:43.293803 20368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:43.294198 20363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:43.294413 20363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:43.295545 20363 hybrid_clock.cc:648] HybridClock initialized: now 1790226463295531 us; error 31 us; skew 500 ppm
I20260924 05:07:43.296708 20363 webserver.cc:460] Webserver started at http://127.0.98.3:37445/ using document root <none> and password file <none>
I20260924 05:07:43.296936 20363 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:43.296979 20363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:43.297091 20363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:43.298169 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "72cc1389c36f3246812eac23f67db3a5"
server_key_iv: "81ee71cbf3d677efc9b8c5e7b0e471c8"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.298436 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "72cc1389c36f3246812eac23f67db3a5"
server_key_iv: "81ee71cbf3d677efc9b8c5e7b0e471c8"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.299494 20363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:43.300069 20377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.300230 20363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:43.300305 20363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal
uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14"
format_stamp: "Formatted at 2026-09-24 05:07:43 on dist-test-slave-m01t"
server_key: "72cc1389c36f3246812eac23f67db3a5"
server_key_iv: "81ee71cbf3d677efc9b8c5e7b0e471c8"
server_key_version: "encryptionkey@0"
I20260924 05:07:43.300366 20363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:43.339859 20363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:43.340164 20363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:43.340291 20363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:43.340504 20363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:07:43.340857 20363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:07:43.340891 20363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.340920 20363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:07:43.340936 20363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:43.346062 20363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46767
I20260924 05:07:43.346102 20490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46767 every 8 connection(s)
I20260924 05:07:43.346412 20363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:07:43.351011 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.347077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58819 (local address 127.0.98.62:33149)
0924 05:07:43.347161 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.347163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.347341 (+ 178us) server_negotiation.cc:407] Connection header received
0924 05:07:43.347791 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.347792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.347801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.347849 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:43.348443 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.348925 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.349861 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.350072 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.350127 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.350779 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:43.350781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:43.350781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:43.350785 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:43.350799 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:43.350803 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:43.350839 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.350929 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.350952 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 05:07:43.351136 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.347040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33149 (local address 127.0.98.3:58819)
0924 05:07:43.347224 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.347258 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.347713 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.347901 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.347904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.347912 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:07:43.348383 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.348387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.348987 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.348987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.349793 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.349796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.350138 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.350139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.350245 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.350664 (+ 419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:07:43.350665 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:07:43.350691 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:07:43.350711 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:07:43.350848 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:07:43.350851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:07:43.350854 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.350898 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.350971 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260924 05:07:43.351267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20363
I20260924 05:07:43.351368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:07:43.351547 392 external_mini_cluster.cc:1428] Setting key 58e639a3e945186cab048609dc57998f
I20260924 05:07:43.351565 20491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33149
I20260924 05:07:43.351652 20491 heartbeater.cc:464] Registering TS with master...
I20260924 05:07:43.351821 20491 heartbeater.cc:511] Master 127.0.98.62:33149 requested a full tablet report, sending...
I20260924 05:07:43.352173 20041 ts_manager.cc:194] Registered new tserver with Master: b0c821a66d7a4bd8816a35ba8c3e1b14 (127.0.98.3:46767)
I20260924 05:07:43.352510 20041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58819
I20260924 05:07:43.352921 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:07:43.356451 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.354168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.98.62:33149)
0924 05:07:43.354214 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.354215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.354304 (+ 89us) server_negotiation.cc:407] Connection header received
0924 05:07:43.354343 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.354344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.354352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.354394 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:43.354507 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.354936 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.355568 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.355731 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.355770 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.356296 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:43.356297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:43.356297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:43.356300 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:43.356311 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:43.356315 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:43.356352 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.356385 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.356403 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:07:43.357829 20041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:07:43.359244 20041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:07:43.364852 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.361101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37743 (local address 127.0.0.1:45394)
0924 05:07:43.361677 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.361686 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.361853 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.362149 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.362150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.362151 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:43.362185 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.362187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.362980 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.362981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.363813 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.363816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.364573 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.364574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.364645 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.364658 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.364736 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.364781 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":28,"thread_start_us":83,"threads_started":1}
I20260924 05:07:43.364868 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.361188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.0.98.1:37743)
0924 05:07:43.361816 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.361820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.361829 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:43.362012 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.362015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.362030 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.362081 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:43.362246 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.362914 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.363896 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.364488 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.364556 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.364647 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.364750 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.364789 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":83,"threads_started":1}
I20260924 05:07:43.365415 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.361089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.0.1:36782)
0924 05:07:43.361295 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.361330 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.361391 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.361600 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.361600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.361602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:43.361666 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.361668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.362937 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.362938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.364238 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.364240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.365198 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.365199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.365261 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.365264 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.365314 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.365348 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":83,"thread_start_us":71,"threads_started":1}
I20260924 05:07:43.365435 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.361287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.0.98.2:40539)
0924 05:07:43.361464 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.361468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.361477 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:43.361510 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.361512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.361526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.361587 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:43.361769 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.362849 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.364330 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.365116 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.365194 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.365278 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.365327 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.365362 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260924 05:07:43.365545 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.361089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46767 (local address 127.0.0.1:39460)
0924 05:07:43.361359 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.361369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.361421 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.361590 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.361593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.361595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:43.361649 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.361650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.363094 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.363094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.364238 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.364240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.365297 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.365298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.365347 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.365350 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.365390 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.365419 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":30,"thread_start_us":38,"threads_started":1}
I20260924 05:07:43.365557 20163 tablet_service.cc:1467] Processing CreateTablet for tablet c1be21f0af1f4e7b9349b95b2eaf546e (DEFAULT_TABLE table=test-table [id=eb33e7ead78745e0877a07b71f0f6244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:43.365731 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.361276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.0.98.3:46767)
0924 05:07:43.361409 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.361414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.361423 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:07:43.361481 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.361483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.361496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.361564 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:43.361731 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.362847 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.364324 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.365135 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.365197 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.365269 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.365455 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.365651 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260924 05:07:43.365939 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1be21f0af1f4e7b9349b95b2eaf546e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:43.366405 20424 tablet_service.cc:1467] Processing CreateTablet for tablet c1be21f0af1f4e7b9349b95b2eaf546e (DEFAULT_TABLE table=test-table [id=eb33e7ead78745e0877a07b71f0f6244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:43.366706 20424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1be21f0af1f4e7b9349b95b2eaf546e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:43.369145 20505 tablet_bootstrap.cc:492] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: Bootstrap starting.
I20260924 05:07:43.369659 20506 tablet_bootstrap.cc:492] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Bootstrap starting.
I20260924 05:07:43.369729 20505 tablet_bootstrap.cc:654] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:43.369699 20294 tablet_service.cc:1467] Processing CreateTablet for tablet c1be21f0af1f4e7b9349b95b2eaf546e (DEFAULT_TABLE table=test-table [id=eb33e7ead78745e0877a07b71f0f6244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:07:43.370028 20294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1be21f0af1f4e7b9349b95b2eaf546e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:43.370040 20505 log.cc:824] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:43.370427 20506 tablet_bootstrap.cc:654] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:43.370756 20506 log.cc:824] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:43.370839 20505 tablet_bootstrap.cc:492] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: No bootstrap required, opened a new log
I20260924 05:07:43.370898 20505 ts_tablet_manager.cc:1397] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:43.371477 20506 tablet_bootstrap.cc:492] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: No bootstrap required, opened a new log
I20260924 05:07:43.371546 20506 ts_tablet_manager.cc:1397] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:07:43.372794 20509 tablet_bootstrap.cc:492] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: Bootstrap starting.
I20260924 05:07:43.373323 20509 tablet_bootstrap.cc:654] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:43.373262 20505 raft_consensus.cc:348] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.373409 20505 raft_consensus.cc:374] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:43.373449 20505 raft_consensus.cc:729] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4550cfaff2b4b5db76aee03fffcf2a8, State: Initialized, Role: FOLLOWER
I20260924 05:07:43.373559 20505 consensus_queue.cc:260] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.373602 20509 log.cc:824] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:43.373824 20505 ts_tablet_manager.cc:1428] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:07:43.373942 20229 heartbeater.cc:503] Master 127.0.98.62:33149 was elected leader, sending a full tablet report...
I20260924 05:07:43.373903 20506 raft_consensus.cc:348] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.374046 20506 raft_consensus.cc:374] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:43.374084 20506 raft_consensus.cc:729] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c821a66d7a4bd8816a35ba8c3e1b14, State: Initialized, Role: FOLLOWER
I20260924 05:07:43.374188 20506 consensus_queue.cc:260] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.374292 20509 tablet_bootstrap.cc:492] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: No bootstrap required, opened a new log
I20260924 05:07:43.374346 20509 ts_tablet_manager.cc:1397] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:07:43.374433 20506 ts_tablet_manager.cc:1428] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:07:43.374447 20491 heartbeater.cc:503] Master 127.0.98.62:33149 was elected leader, sending a full tablet report...
I20260924 05:07:43.376215 20509 raft_consensus.cc:348] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.376327 20509 raft_consensus.cc:374] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:43.376358 20509 raft_consensus.cc:729] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4ea9f5b0284865b0acda67481f0289, State: Initialized, Role: FOLLOWER
I20260924 05:07:43.376485 20509 consensus_queue.cc:260] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.376739 20509 ts_tablet_manager.cc:1428] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:07:43.376780 20360 heartbeater.cc:503] Master 127.0.98.62:33149 was elected leader, sending a full tablet report...
W20260924 05:07:43.453300 20361 tablet.cc:2369] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:07:43.556437 20230 tablet.cc:2369] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:07:43.578840 20512 raft_consensus.cc:482] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:07:43.578958 20512 raft_consensus.cc:504] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.579273 20512 leader_election.cc:288] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4550cfaff2b4b5db76aee03fffcf2a8 (127.0.98.1:37743), 1a4ea9f5b0284865b0acda67481f0289 (127.0.98.2:40539)
I20260924 05:07:43.581883 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.579522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52783 (local address 127.0.98.2:40539)
0924 05:07:43.579580 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.579581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.579589 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:43.579614 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.579615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.579625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.579678 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:43.579816 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.580274 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.581190 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.581539 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.581596 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.581607 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.581779 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.581818 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:07:43.581883 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.579435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.98.3:52783)
0924 05:07:43.579484 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.579498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.579542 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.579680 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.579682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.579683 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:43.579753 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.579755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.580348 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.580349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.581101 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.581103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.581613 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.581614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.581680 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.581683 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.581765 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.581807 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":20}
I20260924 05:07:43.581970 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.579550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33947 (local address 127.0.98.1:37743)
0924 05:07:43.579602 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.579604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.579694 (+ 90us) server_negotiation.cc:407] Connection header received
0924 05:07:43.579749 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.579750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.579758 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.579798 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:07:43.579937 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.580385 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.581282 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.581626 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.581682 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.581692 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.581867 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.581905 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:07:43.582010 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.579439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37743 (local address 127.0.98.3:33947)
0924 05:07:43.579623 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.579632 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:07:43.579689 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:07:43.579810 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:07:43.579811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:07:43.579812 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:07:43.579879 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.579880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.580460 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.580461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.581193 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:07:43.581195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.581703 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.581704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:07:43.581800 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.581803 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:07:43.581848 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:07:43.581885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":10,"thread_start_us":88,"threads_started":1}
I20260924 05:07:43.582223 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1be21f0af1f4e7b9349b95b2eaf546e" candidate_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a4ea9f5b0284865b0acda67481f0289" is_pre_election: true
I20260924 05:07:43.582389 20314 raft_consensus.cc:2454] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c821a66d7a4bd8816a35ba8c3e1b14 in term 0.
I20260924 05:07:43.582355 20183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1be21f0af1f4e7b9349b95b2eaf546e" candidate_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" is_pre_election: true
I20260924 05:07:43.582476 20183 raft_consensus.cc:2454] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c821a66d7a4bd8816a35ba8c3e1b14 in term 0.
I20260924 05:07:43.582630 20381 leader_election.cc:302] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a4ea9f5b0284865b0acda67481f0289, b0c821a66d7a4bd8816a35ba8c3e1b14; no voters:
I20260924 05:07:43.582768 20512 raft_consensus.cc:2781] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:07:43.582821 20512 raft_consensus.cc:482] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:07:43.582842 20512 raft_consensus.cc:3037] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:43.583523 20512 raft_consensus.cc:504] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.583642 20512 leader_election.cc:288] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [CANDIDATE]: Term 1 election: Requested vote from peers e4550cfaff2b4b5db76aee03fffcf2a8 (127.0.98.1:37743), 1a4ea9f5b0284865b0acda67481f0289 (127.0.98.2:40539)
I20260924 05:07:43.583784 20183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1be21f0af1f4e7b9349b95b2eaf546e" candidate_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8"
I20260924 05:07:43.583781 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1be21f0af1f4e7b9349b95b2eaf546e" candidate_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a4ea9f5b0284865b0acda67481f0289"
I20260924 05:07:43.583855 20183 raft_consensus.cc:3037] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:43.583863 20314 raft_consensus.cc:3037] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:43.584503 20183 raft_consensus.cc:2454] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c821a66d7a4bd8816a35ba8c3e1b14 in term 1.
I20260924 05:07:43.584592 20314 raft_consensus.cc:2454] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c821a66d7a4bd8816a35ba8c3e1b14 in term 1.
I20260924 05:07:43.584669 20379 leader_election.cc:302] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c821a66d7a4bd8816a35ba8c3e1b14, e4550cfaff2b4b5db76aee03fffcf2a8; no voters:
I20260924 05:07:43.584748 20512 raft_consensus.cc:2781] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:43.584892 20512 raft_consensus.cc:686] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 1 LEADER]: Becoming Leader. State: Replica: b0c821a66d7a4bd8816a35ba8c3e1b14, State: Running, Role: LEADER
I20260924 05:07:43.584975 20512 consensus_queue.cc:237] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } }
I20260924 05:07:43.585741 20040 catalog_manager.cc:5495] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c821a66d7a4bd8816a35ba8c3e1b14 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c821a66d7a4bd8816a35ba8c3e1b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 } health_report { overall_health: UNKNOWN } } }
W20260924 05:07:43.596884 20492 tablet.cc:2369] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:07:43.645248 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.643215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.0.98.3:46767)
0924 05:07:43.643262 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:43.643264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:43.643272 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:07:43.643306 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:43.643307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:43.643324 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:43.643379 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:07:43.643478 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.643937 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.644637 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:43.644822 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:43.644866 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:43.644968 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:07:43.645057 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:07:43.645106 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:43.645131 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 05:07:43.645144 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:07:43.647013 20314 raft_consensus.cc:1261] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 1 FOLLOWER]: Refusing update from remote peer b0c821a66d7a4bd8816a35ba8c3e1b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:43.647014 20183 raft_consensus.cc:1261] T c1be21f0af1f4e7b9349b95b2eaf546e P e4550cfaff2b4b5db76aee03fffcf2a8 [term 1 FOLLOWER]: Refusing update from remote peer b0c821a66d7a4bd8816a35ba8c3e1b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:07:43.647327 20512 consensus_queue.cc:1035] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4550cfaff2b4b5db76aee03fffcf2a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:43.647469 20515 consensus_queue.cc:1035] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a4ea9f5b0284865b0acda67481f0289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:07:43.649801 20519 mvcc.cc:205] Tried to move back new op lower bound from 7332767595095498752 to 7332767594844676096. Current Snapshot: MvccSnapshot[applied={T|T < 7332767595095498752}]
I20260924 05:07:43.650218 20520 mvcc.cc:205] Tried to move back new op lower bound from 7332767595095498752 to 7332767594844676096. Current Snapshot: MvccSnapshot[applied={T|T < 7332767595095498752}]
I20260924 05:07:43.651647 20041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:43.651734 20041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:07:43.652385 20041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a044c7f3fa074ab0a06c0da209678c01: Sending DeleteTablet for 3 replicas of tablet c1be21f0af1f4e7b9349b95b2eaf546e
I20260924 05:07:43.652689 20163 tablet_service.cc:1514] Processing DeleteTablet for tablet c1be21f0af1f4e7b9349b95b2eaf546e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:43 UTC) from {username='slave'} at 127.0.0.1:45394
I20260924 05:07:43.652906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20101
I20260924 05:07:43.652895 20294 tablet_service.cc:1514] Processing DeleteTablet for tablet c1be21f0af1f4e7b9349b95b2eaf546e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:43 UTC) from {username='slave'} at 127.0.0.1:36782
I20260924 05:07:43.653806 20528 tablet_replica.cc:330] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289: stopping tablet replica
I20260924 05:07:43.653954 20528 raft_consensus.cc:2229] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:07:43.654081 20528 raft_consensus.cc:2258] T c1be21f0af1f4e7b9349b95b2eaf546e P 1a4ea9f5b0284865b0acda67481f0289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:43.654661 20424 tablet_service.cc:1514] Processing DeleteTablet for tablet c1be21f0af1f4e7b9349b95b2eaf546e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:07:43 UTC) from {username='slave'} at 127.0.0.1:39460
I20260924 05:07:43.654990 20529 tablet_replica.cc:330] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: stopping tablet replica
I20260924 05:07:43.655087 20529 raft_consensus.cc:2229] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:07:43.655175 20529 raft_consensus.cc:2258] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:07:43.655586 20529 ts_tablet_manager.cc:1905] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:07:43.657238 20529 ts_tablet_manager.cc:1918] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:07:43.657308 20529 log.cc:1196] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Deleting WAL directory at /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226228698807-392-0/minicluster-data/ts-2/wal/wals/c1be21f0af1f4e7b9349b95b2eaf546e
I20260924 05:07:43.657579 20529 ts_tablet_manager.cc:1939] T c1be21f0af1f4e7b9349b95b2eaf546e P b0c821a66d7a4bd8816a35ba8c3e1b14: Deleting consensus metadata
W20260924 05:07:43.657784 20026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37743 (error 108)
I20260924 05:07:43.658022 20025 catalog_manager.cc:4841] TS b0c821a66d7a4bd8816a35ba8c3e1b14 (127.0.98.3:46767): tablet c1be21f0af1f4e7b9349b95b2eaf546e (table test-table [id=eb33e7ead78745e0877a07b71f0f6244]) successfully deleted
I20260924 05:07:43.658051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20232
I20260924 05:07:43.658680 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.658577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37743 (local address 127.0.0.1:45408)
0924 05:07:43.658617 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.658651 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:07:43.658805 20026 catalog_manager.cc:4584] TS e4550cfaff2b4b5db76aee03fffcf2a8 (127.0.98.1:37743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1be21f0af1f4e7b9349b95b2eaf546e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37743: connect: Connection refused (error 111)
I20260924 05:07:43.661767 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:43.661643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40539 (local address 127.0.0.1:36788)
0924 05:07:43.661696 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:07:43.661743 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:07:43.662412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20363
I20260924 05:07:43.666240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20010
2026-09-24T05:07:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (908 ms)
[----------] 6 tests from EncryptionPolicyTest (8150 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:43 dist-test-slave-m01t krb5kdc[20533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:43 dist-test-slave-m01t krb5kdc[20533](info): set up 2 sockets
Sep 24 05:07:43 dist-test-slave-m01t krb5kdc[20533](info): commencing operation
krb5kdc: starting...
W20260924 05:07:44.617343 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:44 dist-test-slave-m01t krb5kdc[20533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:44.722512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:44.798671 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:44.798854 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:44.798883 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:44.799973 20546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:44.800014 20546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:44.800033 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:44.800048 20546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:44.800062 20546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:44.801448 20546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:44.801664 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:44.801867 20546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:44.806185 20551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:44.806152 20552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:44.806156 20554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:44.806202 20546 server_base.cc:1034] running on GCE node
I20260924 05:07:44.806545 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:44.806790 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:44.807920 20546 hybrid_clock.cc:648] HybridClock initialized: now 1790226464807906 us; error 28 us; skew 500 ppm
Sep 24 05:07:44 dist-test-slave-m01t krb5kdc[20533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:44.814713 20546 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:44.815099 20546 webserver.cc:460] Webserver started at http://127.0.0.1:35915/ using document root <none> and password file <none>
I20260924 05:07:44.815291 20546 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:44.815336 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:44.815454 20546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:44.816398 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "30cb0a3bc1ba429d9c07b62f8266456a"
format_stamp: "Formatted at 2026-09-24 05:07:44 on dist-test-slave-m01t"
server_key: "5ac1307bbdde0a10ba58865e14d248ef"
server_key_iv: "8f3007fd5733fb28a48d2bc7f4fbe712"
server_key_version: "encryptionkey@0"
I20260924 05:07:44.816707 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "30cb0a3bc1ba429d9c07b62f8266456a"
format_stamp: "Formatted at 2026-09-24 05:07:44 on dist-test-slave-m01t"
server_key: "5ac1307bbdde0a10ba58865e14d248ef"
server_key_iv: "8f3007fd5733fb28a48d2bc7f4fbe712"
server_key_version: "encryptionkey@0"
I20260924 05:07:44.817916 20546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:07:44.818548 20561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:44.818748 20546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:44.818835 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "30cb0a3bc1ba429d9c07b62f8266456a"
format_stamp: "Formatted at 2026-09-24 05:07:44 on dist-test-slave-m01t"
server_key: "5ac1307bbdde0a10ba58865e14d248ef"
server_key_iv: "8f3007fd5733fb28a48d2bc7f4fbe712"
server_key_version: "encryptionkey@0"
I20260924 05:07:44.818897 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:44.830683 20546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:44.832392 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:44.832532 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:44.836316 20546 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41425
I20260924 05:07:44.836349 20613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41425 every 8 connection(s)
I20260924 05:07:44.836691 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:44.837522 20614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:44.839748 20614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a: Bootstrap starting.
I20260924 05:07:44.840293 20614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:44.840538 20614 log.cc:824] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:44.841161 20614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a: No bootstrap required, opened a new log
I20260924 05:07:44.842711 20614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41425 } }
I20260924 05:07:44.842844 20614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:44.842875 20614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cb0a3bc1ba429d9c07b62f8266456a, State: Initialized, Role: FOLLOWER
I20260924 05:07:44.843000 20614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41425 } }
I20260924 05:07:44.843076 20614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:44.843107 20614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:44.843163 20614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:44.843631 20614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41425 } }
I20260924 05:07:44.843721 20614 leader_election.cc:302] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30cb0a3bc1ba429d9c07b62f8266456a; no voters:
I20260924 05:07:44.843863 20614 leader_election.cc:288] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:44.843895 20617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:44.844018 20617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [term 1 LEADER]: Becoming Leader. State: Replica: 30cb0a3bc1ba429d9c07b62f8266456a, State: Running, Role: LEADER
I20260924 05:07:44.844092 20614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:44.844101 20617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41425 } }
I20260924 05:07:44.844550 20619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30cb0a3bc1ba429d9c07b62f8266456a. Latest consensus state: current_term: 1 leader_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41425 } } }
I20260924 05:07:44.844628 20619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:44.844918 20618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30cb0a3bc1ba429d9c07b62f8266456a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41425 } } }
I20260924 05:07:44.844976 20626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:44.844990 20618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:44.845427 20626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:44.846679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20546
I20260924 05:07:44.846807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:44.847004 392 external_mini_cluster.cc:1428] Setting key 70eb1a5197f4203a9072ac743ef862c5
I20260924 05:07:44.847234 20626 catalog_manager.cc:1347] Generated new cluster ID: d59f5522edf64dac804781e33b5b2d5b
I20260924 05:07:44.847280 20626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:07:44 dist-test-slave-m01t krb5kdc[20533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:07:44.861858 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:44.847570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42893 (local address 127.0.0.1:41425)
0924 05:07:44.853834 (+ 6264us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:44.853842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:44.853867 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:07:44.853952 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:44.853957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:44.854007 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:44.854123 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:44.854731 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:44.855614 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:44.856361 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:44.856746 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:44.856834 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:44.858973 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:44.858977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:44.858977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:44.858986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:44.860421 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:44.860779 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:44.860781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:44.860781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:44.860856 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:44.861055 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:44.861056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:44.861056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:44.861271 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:44.861395 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:44.861530 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 05:07:44.861622 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6221,"thread_start_us":68,"threads_started":1}
I20260924 05:07:44.862658 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:44.874981 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:44.865934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37421 (local address 127.0.0.1:41425)
0924 05:07:44.865964 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:44.865966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:44.869848 (+ 3882us) server_negotiation.cc:407] Connection header received
0924 05:07:44.869898 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:44.869899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:44.869907 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:44.869952 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:44.870388 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:44.870852 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:44.871500 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:44.871695 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:44.871745 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:44.872746 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:44.872747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:44.872747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:44.872753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:44.873981 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:44.874297 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:44.874298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:44.874298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:44.874342 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:44.874529 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:44.874530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:44.874530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:44.874621 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:44.874714 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:44.874854 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:07:44.874906 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:07:44.876946 20626 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:44.877403 20626 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:44.881286 20626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30cb0a3bc1ba429d9c07b62f8266456a: Generated new TSK 0
I20260924 05:07:44.881421 20626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:44.886803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20546
W20260924 05:07:44.890533 20531 connection.cc:535] client connection to 127.0.0.1:41425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41425): Connection reset by peer (error 104)
2026-09-24T05:07:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:44 dist-test-slave-m01t krb5kdc[20645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:44 dist-test-slave-m01t krb5kdc[20645](info): set up 2 sockets
Sep 24 05:07:44 dist-test-slave-m01t krb5kdc[20645](info): commencing operation
krb5kdc: starting...
W20260924 05:07:45.833500 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:45 dist-test-slave-m01t krb5kdc[20645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:45.946123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:46.024501 20658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:46.024706 20658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:46.024729 20658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:46.025911 20658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:46.025952 20658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:46.025964 20658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:46.025977 20658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:46.026000 20658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:46.027496 20658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39543
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:46.027731 20658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:46.027941 20658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:46.032574 20664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:46.032655 20666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:46.032677 20658 server_base.cc:1034] running on GCE node
W20260924 05:07:46.032578 20663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:46.032979 20658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:46.033233 20658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:46.034366 20658 hybrid_clock.cc:648] HybridClock initialized: now 1790226466034338 us; error 45 us; skew 500 ppm
Sep 24 05:07:46 dist-test-slave-m01t krb5kdc[20645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:46.036723 20658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:46.037130 20658 webserver.cc:460] Webserver started at http://127.0.0.1:39547/ using document root <none> and password file <none>
I20260924 05:07:46.037343 20658 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:46.037391 20658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:46.037535 20658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:46.038602 20658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "36993122ec8442819941ce9465e664aa"
format_stamp: "Formatted at 2026-09-24 05:07:46 on dist-test-slave-m01t"
server_key: "d03fea43987b1debd556d862c3d62373"
server_key_iv: "06b0228dc5344d1b262ca9d215099b95"
server_key_version: "encryptionkey@0"
I20260924 05:07:46.038940 20658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "36993122ec8442819941ce9465e664aa"
format_stamp: "Formatted at 2026-09-24 05:07:46 on dist-test-slave-m01t"
server_key: "d03fea43987b1debd556d862c3d62373"
server_key_iv: "06b0228dc5344d1b262ca9d215099b95"
server_key_version: "encryptionkey@0"
I20260924 05:07:46.040166 20658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:46.040817 20673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:46.040972 20658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:46.041051 20658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "36993122ec8442819941ce9465e664aa"
format_stamp: "Formatted at 2026-09-24 05:07:46 on dist-test-slave-m01t"
server_key: "d03fea43987b1debd556d862c3d62373"
server_key_iv: "06b0228dc5344d1b262ca9d215099b95"
server_key_version: "encryptionkey@0"
I20260924 05:07:46.041117 20658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:46.056109 20658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:46.057648 20658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:46.057787 20658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:46.061187 20658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39543
I20260924 05:07:46.061246 20725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39543 every 8 connection(s)
I20260924 05:07:46.061511 20658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:46.062325 20726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:46.064433 20726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa: Bootstrap starting.
I20260924 05:07:46.064985 20726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:46.065238 20726 log.cc:824] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:46.065876 20726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa: No bootstrap required, opened a new log
I20260924 05:07:46.067351 20726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36993122ec8442819941ce9465e664aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39543 } }
I20260924 05:07:46.067479 20726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:46.067509 20726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36993122ec8442819941ce9465e664aa, State: Initialized, Role: FOLLOWER
I20260924 05:07:46.067656 20726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36993122ec8442819941ce9465e664aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39543 } }
I20260924 05:07:46.067734 20726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:46.067767 20726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:46.067813 20726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:46.068287 20726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36993122ec8442819941ce9465e664aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39543 } }
I20260924 05:07:46.068372 20726 leader_election.cc:302] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36993122ec8442819941ce9465e664aa; no voters:
I20260924 05:07:46.068514 20726 leader_election.cc:288] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:46.068547 20729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:46.068698 20729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [term 1 LEADER]: Becoming Leader. State: Replica: 36993122ec8442819941ce9465e664aa, State: Running, Role: LEADER
I20260924 05:07:46.068755 20726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:46.068780 20729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36993122ec8442819941ce9465e664aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39543 } }
I20260924 05:07:46.069120 20731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36993122ec8442819941ce9465e664aa. Latest consensus state: current_term: 1 leader_uuid: "36993122ec8442819941ce9465e664aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36993122ec8442819941ce9465e664aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39543 } } }
I20260924 05:07:46.069233 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36993122ec8442819941ce9465e664aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36993122ec8442819941ce9465e664aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39543 } } }
I20260924 05:07:46.069254 20731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:46.069314 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:46.069676 20738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:46.070094 20738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:46.070281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20658
I20260924 05:07:46.070397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:46.070595 392 external_mini_cluster.cc:1428] Setting key fa15c069b25137c1ff7cf248e9fc0959
I20260924 05:07:46.071834 20738 catalog_manager.cc:1347] Generated new cluster ID: 24731faf6bec4e20bc293c7785dd92e1
I20260924 05:07:46.071875 20738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:46.077589 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:46.071142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36413 (local address 127.0.0.1:39543)
0924 05:07:46.071888 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:46.071896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:46.071925 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:07:46.072029 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:46.072035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:46.072086 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:46.072223 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:46.072458 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:46.073943 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:46.074992 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:46.075625 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:46.075766 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:46.076604 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:46.076607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:46.076608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:46.076869 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:46.076927 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:46.076942 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:46.077008 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:46.077229 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 05:07:46.077255 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":83,"threads_started":1}
I20260924 05:07:46.078611 20738 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:46.079066 20738 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:46.079763 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:46.081804 20738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36993122ec8442819941ce9465e664aa: Generated new TSK 0
I20260924 05:07:46.081939 20738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:46.084321 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:46.081307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57609 (local address 127.0.0.1:39543)
0924 05:07:46.081342 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:46.081344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:46.081351 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:07:46.081388 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:46.081389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:46.081399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:46.081450 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:46.081558 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:46.082228 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:46.083302 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:46.083504 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:46.083550 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:46.084127 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:46.084129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:46.084129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:46.084132 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:46.084143 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:46.084148 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:46.084184 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:46.084267 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 05:07:46.084277 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:07:46.085938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20658
W20260924 05:07:46.089413 20643 connection.cc:535] client connection to 127.0.0.1:39543 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39543): Connection reset by peer (error 104)
2026-09-24T05:07:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1198 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:46 dist-test-slave-m01t krb5kdc[20757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:46 dist-test-slave-m01t krb5kdc[20757](info): set up 2 sockets
Sep 24 05:07:46 dist-test-slave-m01t krb5kdc[20757](info): commencing operation
krb5kdc: starting...
W20260924 05:07:47.025576 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:47 dist-test-slave-m01t krb5kdc[20757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:47.135825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:47.211798 20770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:47.211992 20770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:47.212013 20770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:47.213116 20770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:47.213160 20770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:47.213171 20770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:47.213184 20770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:47.213207 20770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:47.214622 20770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:47.214846 20770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:47.215037 20770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:47.219671 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:47.219661 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:47.219714 20770 server_base.cc:1034] running on GCE node
W20260924 05:07:47.219674 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:47.220037 20770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:47.220317 20770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:47.221455 20770 hybrid_clock.cc:648] HybridClock initialized: now 1790226467221444 us; error 26 us; skew 500 ppm
Sep 24 05:07:47 dist-test-slave-m01t krb5kdc[20757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:47.223764 20770 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:47.224210 20770 webserver.cc:460] Webserver started at http://127.0.0.1:45631/ using document root <none> and password file <none>
I20260924 05:07:47.224419 20770 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:47.224467 20770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:47.224591 20770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:47.225605 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "ca2743affc6545498eb18cfa57adf1d6"
format_stamp: "Formatted at 2026-09-24 05:07:47 on dist-test-slave-m01t"
server_key: "7cbaff137fd2f33933deda5403132d38"
server_key_iv: "3383b4d74e66f1f47957608538cac753"
server_key_version: "encryptionkey@0"
I20260924 05:07:47.225991 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca2743affc6545498eb18cfa57adf1d6"
format_stamp: "Formatted at 2026-09-24 05:07:47 on dist-test-slave-m01t"
server_key: "7cbaff137fd2f33933deda5403132d38"
server_key_iv: "3383b4d74e66f1f47957608538cac753"
server_key_version: "encryptionkey@0"
I20260924 05:07:47.227203 20770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:07:47.227819 20785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:47.227959 20770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:47.228011 20770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "ca2743affc6545498eb18cfa57adf1d6"
format_stamp: "Formatted at 2026-09-24 05:07:47 on dist-test-slave-m01t"
server_key: "7cbaff137fd2f33933deda5403132d38"
server_key_iv: "3383b4d74e66f1f47957608538cac753"
server_key_version: "encryptionkey@0"
I20260924 05:07:47.228088 20770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:47.231393 20770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:47.233428 20770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:47.233551 20770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:47.237015 20770 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33195
I20260924 05:07:47.237046 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33195 every 8 connection(s)
I20260924 05:07:47.237354 20770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:47.238334 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:47.239823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20770
I20260924 05:07:47.239923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:47.240096 392 external_mini_cluster.cc:1428] Setting key 5690d53955f8d91319f4f07e29390712
I20260924 05:07:47.240666 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6: Bootstrap starting.
I20260924 05:07:47.241341 20838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:47.241670 20838 log.cc:824] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:47.242378 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6: No bootstrap required, opened a new log
I20260924 05:07:47.242532 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:47.240592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56907 (local address 127.0.0.1:33195)
0924 05:07:47.240783 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:47.240793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:47.240823 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:07:47.241360 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:47.241370 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:47.241411 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:47.241562 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:47.242120 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:47.242124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:47.242124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:47.242127 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:47.242168 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:47.242178 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:47.242243 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:47.242310 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:07:47.242341 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260924 05:07:47.243522 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:47.244689 20838 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2743affc6545498eb18cfa57adf1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260924 05:07:47.244860 20838 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:47.244906 20838 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2743affc6545498eb18cfa57adf1d6, State: Initialized, Role: FOLLOWER
I20260924 05:07:47.245019 20838 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2743affc6545498eb18cfa57adf1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260924 05:07:47.245110 20838 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:47.245141 20838 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:47.245189 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:47.245857 20838 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2743affc6545498eb18cfa57adf1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260924 05:07:47.245967 20838 leader_election.cc:302] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2743affc6545498eb18cfa57adf1d6; no voters:
I20260924 05:07:47.246070 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:47.245042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36717 (local address 127.0.0.1:33195)
0924 05:07:47.245072 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:47.245074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:47.245229 (+ 155us) server_negotiation.cc:407] Connection header received
0924 05:07:47.245284 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:47.245286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:47.245290 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:47.245353 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:47.245863 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:47.245864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:47.245865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:47.245869 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:47.245882 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:47.245886 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:47.245948 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:47.245999 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 05:07:47.246010 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 05:07:47.246165 20838 leader_election.cc:288] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:47.246186 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:47.246309 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [term 1 LEADER]: Becoming Leader. State: Replica: ca2743affc6545498eb18cfa57adf1d6, State: Running, Role: LEADER
I20260924 05:07:47.246425 20838 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:47.246414 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2743affc6545498eb18cfa57adf1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } }
I20260924 05:07:47.246771 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2743affc6545498eb18cfa57adf1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2743affc6545498eb18cfa57adf1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } } }
I20260924 05:07:47.246783 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2743affc6545498eb18cfa57adf1d6. Latest consensus state: current_term: 1 leader_uuid: "ca2743affc6545498eb18cfa57adf1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2743affc6545498eb18cfa57adf1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33195 } } }
I20260924 05:07:47.246912 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:47.247090 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:47.247328 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:47.247795 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:47.249148 20856 catalog_manager.cc:1347] Generated new cluster ID: 40adfdd117b64bf4b4f0f23460cdb5a7
I20260924 05:07:47.249184 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:47.268447 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:47.268599 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:47.276304 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2743affc6545498eb18cfa57adf1d6: Generated new TSK 0
I20260924 05:07:47.276480 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:47.284814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20770
2026-09-24T05:07:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:47 dist-test-slave-m01t krb5kdc[20869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:47 dist-test-slave-m01t krb5kdc[20869](info): set up 2 sockets
Sep 24 05:07:47 dist-test-slave-m01t krb5kdc[20869](info): commencing operation
krb5kdc: starting...
W20260924 05:07:48.237593 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:48 dist-test-slave-m01t krb5kdc[20869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:48.345607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:48.422576 20882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:48.422786 20882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:48.422809 20882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:48.423910 20882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:48.423943 20882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:48.423955 20882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:48.423966 20882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:48.423976 20882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:48.425383 20882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:48.425580 20882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:48.425797 20882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:48.430259 20887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:48.430248 20888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:48.430488 20882 server_base.cc:1034] running on GCE node
W20260924 05:07:48.430275 20890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:48.430753 20882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:48.431044 20882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:48.432197 20882 hybrid_clock.cc:648] HybridClock initialized: now 1790226468432190 us; error 35 us; skew 500 ppm
Sep 24 05:07:48 dist-test-slave-m01t krb5kdc[20869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:48.434463 20882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:48.434855 20882 webserver.cc:460] Webserver started at http://127.0.0.1:45597/ using document root <none> and password file <none>
I20260924 05:07:48.435066 20882 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:48.435114 20882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:48.435236 20882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:48.436247 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "212a66bcb69e421db5bf2cceaa1d1b74"
format_stamp: "Formatted at 2026-09-24 05:07:48 on dist-test-slave-m01t"
server_key: "404fbeb63f92bd1f571816d775a35bb7"
server_key_iv: "95d6fc26c4c88e725cedf34b6fd620f9"
server_key_version: "encryptionkey@0"
I20260924 05:07:48.436599 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "212a66bcb69e421db5bf2cceaa1d1b74"
format_stamp: "Formatted at 2026-09-24 05:07:48 on dist-test-slave-m01t"
server_key: "404fbeb63f92bd1f571816d775a35bb7"
server_key_iv: "95d6fc26c4c88e725cedf34b6fd620f9"
server_key_version: "encryptionkey@0"
I20260924 05:07:48.437829 20882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:07:48.438477 20897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:48.438624 20882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:48.438700 20882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "212a66bcb69e421db5bf2cceaa1d1b74"
format_stamp: "Formatted at 2026-09-24 05:07:48 on dist-test-slave-m01t"
server_key: "404fbeb63f92bd1f571816d775a35bb7"
server_key_iv: "95d6fc26c4c88e725cedf34b6fd620f9"
server_key_version: "encryptionkey@0"
I20260924 05:07:48.438767 20882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:48.449918 20882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:48.451462 20882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:48.451582 20882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:48.455025 20882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43493
I20260924 05:07:48.455066 20949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43493 every 8 connection(s)
I20260924 05:07:48.455339 20882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:48.456095 20950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:48.458242 20950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74: Bootstrap starting.
I20260924 05:07:48.458737 20950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:48.458972 20950 log.cc:824] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:48.459556 20950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74: No bootstrap required, opened a new log
I20260924 05:07:48.459833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20882
I20260924 05:07:48.459949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:48.460103 392 external_mini_cluster.cc:1428] Setting key 6a65949c15b897357d323cfd5f89719d
I20260924 05:07:48.461618 20950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260924 05:07:48.461802 20950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:48.461846 20950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212a66bcb69e421db5bf2cceaa1d1b74, State: Initialized, Role: FOLLOWER
I20260924 05:07:48.461967 20950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260924 05:07:48.462045 20950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:48.462077 20950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:48.462121 20950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:48.462607 20950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260924 05:07:48.462692 20950 leader_election.cc:302] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212a66bcb69e421db5bf2cceaa1d1b74; no voters:
I20260924 05:07:48.462828 20950 leader_election.cc:288] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:48.462870 20955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:48.463007 20955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [term 1 LEADER]: Becoming Leader. State: Replica: 212a66bcb69e421db5bf2cceaa1d1b74, State: Running, Role: LEADER
I20260924 05:07:48.463064 20950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:48.463107 20955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260924 05:07:48.463421 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212a66bcb69e421db5bf2cceaa1d1b74. Latest consensus state: current_term: 1 leader_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } } }
I20260924 05:07:48.463639 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:48.463563 20956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212a66bcb69e421db5bf2cceaa1d1b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } } }
I20260924 05:07:48.464042 20964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:48.464057 20956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:48.464557 20964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:48.466670 20964 catalog_manager.cc:1347] Generated new cluster ID: ff9f3ef0d6904dbba588527de8ec3daa
I20260924 05:07:48.466710 20964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:07:48 dist-test-slave-m01t krb5kdc[20869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:07:48.471024 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:48.460548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55589 (local address 127.0.0.1:43493)
0924 05:07:48.460705 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:48.460715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:48.460739 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:07:48.461232 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:48.461238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:48.461280 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:48.461386 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:48.462155 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:48.463562 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:48.464608 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:48.465252 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:48.465351 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:48.467748 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:48.467752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:48.467752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:48.467762 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:48.469520 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:48.469926 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:48.469929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:48.469929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:48.470026 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:48.470218 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:48.470218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:48.470218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:48.470468 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:48.470590 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:48.470711 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 05:07:48.470826 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260924 05:07:48.472057 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:48.479658 20964 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:48.480118 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:48.473462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48145 (local address 127.0.0.1:43493)
0924 05:07:48.473491 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:48.473493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:48.473536 (+ 43us) server_negotiation.cc:407] Connection header received
0924 05:07:48.473602 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:48.473603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:48.473610 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:48.473655 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:48.474247 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:48.474919 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:48.475905 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:48.476205 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:48.476255 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:48.477472 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:48.477473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:48.477473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:48.477479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:48.478900 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:48.479261 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:48.479262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:48.479262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:48.479315 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:48.479562 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:48.479563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:48.479563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:48.479695 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:48.479782 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:48.479974 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 05:07:48.480041 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:07:48.480261 20964 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:48.484130 20964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 212a66bcb69e421db5bf2cceaa1d1b74: Generated new TSK 0
I20260924 05:07:48.484297 20964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:48.487461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20882
2026-09-24T05:07:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:48 dist-test-slave-m01t krb5kdc[20981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:48 dist-test-slave-m01t krb5kdc[20981](info): set up 2 sockets
Sep 24 05:07:48 dist-test-slave-m01t krb5kdc[20981](info): commencing operation
krb5kdc: starting...
W20260924 05:07:49.425565 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:49 dist-test-slave-m01t krb5kdc[20981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:49.530411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:49.608549 20994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:49.608748 20994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:49.608778 20994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:49.609918 20994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:49.609956 20994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:49.609975 20994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:49.609990 20994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:49.610004 20994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:49.611420 20994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:49.611647 20994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:49.611840 20994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:49.616425 21000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:49.616410 20999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:49.616408 21002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:49.616638 20994 server_base.cc:1034] running on GCE node
I20260924 05:07:49.616853 20994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:49.617111 20994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:49.618244 20994 hybrid_clock.cc:648] HybridClock initialized: now 1790226469618224 us; error 34 us; skew 500 ppm
Sep 24 05:07:49 dist-test-slave-m01t krb5kdc[20981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:49.620417 20994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:49.620810 20994 webserver.cc:460] Webserver started at http://127.0.0.1:40545/ using document root <none> and password file <none>
I20260924 05:07:49.621011 20994 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:49.621055 20994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:49.621182 20994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:49.622234 20994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "495482597351459b81026b4a85ff8101"
format_stamp: "Formatted at 2026-09-24 05:07:49 on dist-test-slave-m01t"
server_key: "44ec2d71249ff3865248748ece3fe9cd"
server_key_iv: "e86da1466c20bc425a2ecff9e8c041e3"
server_key_version: "encryptionkey@0"
I20260924 05:07:49.622560 20994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "495482597351459b81026b4a85ff8101"
format_stamp: "Formatted at 2026-09-24 05:07:49 on dist-test-slave-m01t"
server_key: "44ec2d71249ff3865248748ece3fe9cd"
server_key_iv: "e86da1466c20bc425a2ecff9e8c041e3"
server_key_version: "encryptionkey@0"
I20260924 05:07:49.623749 20994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:49.624436 21009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:49.624593 20994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:49.624678 20994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "495482597351459b81026b4a85ff8101"
format_stamp: "Formatted at 2026-09-24 05:07:49 on dist-test-slave-m01t"
server_key: "44ec2d71249ff3865248748ece3fe9cd"
server_key_iv: "e86da1466c20bc425a2ecff9e8c041e3"
server_key_version: "encryptionkey@0"
I20260924 05:07:49.624742 20994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:49.641543 20994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:49.643332 20994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:49.643463 20994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:49.647049 20994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40915
I20260924 05:07:49.647097 21061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40915 every 8 connection(s)
I20260924 05:07:49.647404 20994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:49.648175 21062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:49.650355 21062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101: Bootstrap starting.
I20260924 05:07:49.650846 21062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:49.651073 21062 log.cc:824] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:49.651651 21062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101: No bootstrap required, opened a new log
I20260924 05:07:49.653112 21062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495482597351459b81026b4a85ff8101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260924 05:07:49.653244 21062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:49.653263 21062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495482597351459b81026b4a85ff8101, State: Initialized, Role: FOLLOWER
I20260924 05:07:49.653371 21062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495482597351459b81026b4a85ff8101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260924 05:07:49.653443 21062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:49.653475 21062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:49.653508 21062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:49.653981 21062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495482597351459b81026b4a85ff8101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260924 05:07:49.654068 21062 leader_election.cc:302] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495482597351459b81026b4a85ff8101; no voters:
I20260924 05:07:49.654181 21062 leader_election.cc:288] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:49.654218 21065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:49.654386 21062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:49.654399 21065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [term 1 LEADER]: Becoming Leader. State: Replica: 495482597351459b81026b4a85ff8101, State: Running, Role: LEADER
I20260924 05:07:49.654539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 20994
I20260924 05:07:49.654583 21065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495482597351459b81026b4a85ff8101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260924 05:07:49.654665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:49.654826 392 external_mini_cluster.cc:1428] Setting key 6ec6075b0eb5d9ac78625ea4e415c3e7
I20260924 05:07:49.655135 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495482597351459b81026b4a85ff8101. Latest consensus state: current_term: 1 leader_uuid: "495482597351459b81026b4a85ff8101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495482597351459b81026b4a85ff8101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } } }
I20260924 05:07:49.655279 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:49.655490 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495482597351459b81026b4a85ff8101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495482597351459b81026b4a85ff8101" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } } }
I20260924 05:07:49.655609 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:49.655867 21082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:49.656435 21082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:49.657691 21082 catalog_manager.cc:1347] Generated new cluster ID: 2745d535e70b401492e4cda8703f7b0b
I20260924 05:07:49.657754 21082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:49.661027 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:49.655380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55821 (local address 127.0.0.1:40915)
0924 05:07:49.655691 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:49.655701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:49.655898 (+ 197us) server_negotiation.cc:407] Connection header received
0924 05:07:49.656085 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:49.656093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:49.656146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:49.656259 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:49.656607 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:49.657963 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:49.659036 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:49.659647 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:49.659756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:49.660390 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:49.660394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:49.660394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:49.660591 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:49.660630 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:49.660644 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:49.660704 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:49.660765 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 05:07:49.660826 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":62,"threads_started":1}
I20260924 05:07:49.661953 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:49.663625 21082 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:49.664105 21082 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:49.668967 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:49.663403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33837 (local address 127.0.0.1:40915)
0924 05:07:49.663432 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:49.663433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:49.663611 (+ 178us) server_negotiation.cc:407] Connection header received
0924 05:07:49.665844 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:49.665845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:49.665855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:49.665900 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:49.666031 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:49.666730 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:49.667764 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:49.668040 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:49.668087 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:49.668756 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:49.668757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:49.668757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:49.668763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:49.668776 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:49.668779 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:49.668825 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:49.668877 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 05:07:49.668899 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:07:49.679682 21082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495482597351459b81026b4a85ff8101: Generated new TSK 0
I20260924 05:07:49.679822 21082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:49.681699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 20994
2026-09-24T05:07:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:49 dist-test-slave-m01t krb5kdc[21093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:49 dist-test-slave-m01t krb5kdc[21093](info): set up 2 sockets
Sep 24 05:07:49 dist-test-slave-m01t krb5kdc[21093](info): commencing operation
krb5kdc: starting...
W20260924 05:07:50.617374 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:50 dist-test-slave-m01t krb5kdc[21093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:50.719578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:50.796938 21106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:50.797142 21106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:50.797173 21106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:50.798336 21106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:50.798378 21106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:50.798396 21106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:50.798412 21106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:50.798426 21106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:50.799849 21106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:50.800084 21106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:50.800287 21106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:50.804682 21112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:50.804795 21106 server_base.cc:1034] running on GCE node
W20260924 05:07:50.804764 21114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:50.804699 21111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:50.805110 21106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:50.805372 21106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:50.806506 21106 hybrid_clock.cc:648] HybridClock initialized: now 1790226470806488 us; error 29 us; skew 500 ppm
Sep 24 05:07:50 dist-test-slave-m01t krb5kdc[21093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:50.816421 21106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:50.816855 21106 webserver.cc:460] Webserver started at http://127.0.0.1:38497/ using document root <none> and password file <none>
I20260924 05:07:50.817052 21106 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:50.817091 21106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:50.817181 21106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:50.818228 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "a4f41462bb974fb9adbf13ac534933e3"
format_stamp: "Formatted at 2026-09-24 05:07:50 on dist-test-slave-m01t"
server_key: "c4159b0700c45392ab9c1abdc995b025"
server_key_iv: "834e3e724420389bb8893f69e105aa02"
server_key_version: "encryptionkey@0"
I20260924 05:07:50.818523 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4f41462bb974fb9adbf13ac534933e3"
format_stamp: "Formatted at 2026-09-24 05:07:50 on dist-test-slave-m01t"
server_key: "c4159b0700c45392ab9c1abdc995b025"
server_key_iv: "834e3e724420389bb8893f69e105aa02"
server_key_version: "encryptionkey@0"
I20260924 05:07:50.819671 21106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:50.820289 21121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:50.820451 21106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:50.820528 21106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "a4f41462bb974fb9adbf13ac534933e3"
format_stamp: "Formatted at 2026-09-24 05:07:50 on dist-test-slave-m01t"
server_key: "c4159b0700c45392ab9c1abdc995b025"
server_key_iv: "834e3e724420389bb8893f69e105aa02"
server_key_version: "encryptionkey@0"
I20260924 05:07:50.820582 21106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:50.823792 21106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:50.825822 21106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:50.825928 21106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:50.829368 21106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35827
I20260924 05:07:50.829401 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35827 every 8 connection(s)
I20260924 05:07:50.829700 21106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:50.830651 21174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:50.832870 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3: Bootstrap starting.
I20260924 05:07:50.833115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21106
I20260924 05:07:50.833215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:50.833382 392 external_mini_cluster.cc:1428] Setting key ee3fb12d2aee79b881b63097e3bf9a0f
I20260924 05:07:50.833441 21174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:50.833748 21174 log.cc:824] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:50.834465 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3: No bootstrap required, opened a new log
I20260924 05:07:50.835721 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:50.833848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42241 (local address 127.0.0.1:35827)
0924 05:07:50.834027 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:50.834037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:50.834070 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:07:50.834544 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:50.834553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:50.834591 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:50.834725 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:50.835322 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:50.835325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:50.835325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:50.835329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:50.835363 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:50.835373 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:50.835434 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:50.835499 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:07:50.835530 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260924 05:07:50.836560 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:50.836633 21174 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f41462bb974fb9adbf13ac534933e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260924 05:07:50.836786 21174 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:50.836819 21174 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f41462bb974fb9adbf13ac534933e3, State: Initialized, Role: FOLLOWER
I20260924 05:07:50.836926 21174 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f41462bb974fb9adbf13ac534933e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260924 05:07:50.837003 21174 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:50.837037 21174 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:50.837083 21174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:50.837821 21174 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f41462bb974fb9adbf13ac534933e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260924 05:07:50.837929 21174 leader_election.cc:302] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4f41462bb974fb9adbf13ac534933e3; no voters:
I20260924 05:07:50.838076 21174 leader_election.cc:288] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:50.838111 21183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:50.838279 21183 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [term 1 LEADER]: Becoming Leader. State: Replica: a4f41462bb974fb9adbf13ac534933e3, State: Running, Role: LEADER
I20260924 05:07:50.838315 21174 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:50.838387 21183 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f41462bb974fb9adbf13ac534933e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260924 05:07:50.838737 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4f41462bb974fb9adbf13ac534933e3. Latest consensus state: current_term: 1 leader_uuid: "a4f41462bb974fb9adbf13ac534933e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f41462bb974fb9adbf13ac534933e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } } }
I20260924 05:07:50.838840 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:50.838967 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4f41462bb974fb9adbf13ac534933e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f41462bb974fb9adbf13ac534933e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } } }
I20260924 05:07:50.839093 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:50.839288 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:50.838209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35577 (local address 127.0.0.1:35827)
0924 05:07:50.838247 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:50.838249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:50.838373 (+ 124us) server_negotiation.cc:407] Connection header received
0924 05:07:50.838470 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:50.838470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:50.838475 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:50.838519 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:50.839088 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:50.839089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:50.839090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:50.839094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:50.839107 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:50.839111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:50.839153 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:50.839228 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 05:07:50.839241 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 05:07:50.840111 21200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:07:50.840152 21200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:07:50.840207 21191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:50.840358 21191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:50.841624 21191 catalog_manager.cc:1347] Generated new cluster ID: 3ba1af0f32af4688816e85a8c4783965
I20260924 05:07:50.841660 21191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:50.862272 21191 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:50.862414 21191 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:50.865648 21191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4f41462bb974fb9adbf13ac534933e3: Generated new TSK 0
I20260924 05:07:50.865820 21191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:50.868503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21106
2026-09-24T05:07:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:50 dist-test-slave-m01t krb5kdc[21205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:50 dist-test-slave-m01t krb5kdc[21205](info): set up 2 sockets
Sep 24 05:07:50 dist-test-slave-m01t krb5kdc[21205](info): commencing operation
krb5kdc: starting...
W20260924 05:07:51.805387 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:51 dist-test-slave-m01t krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:51.912278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:51.989694 21218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:51.989913 21218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:51.989943 21218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:51.991073 21218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:51.991113 21218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:51.991130 21218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:51.991146 21218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:51.991160 21218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:51.992578 21218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:51.992803 21218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:51.992985 21218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:51.997428 21223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:51.997440 21226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:51.997438 21224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:51.997581 21218 server_base.cc:1034] running on GCE node
I20260924 05:07:51.997879 21218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:51.998147 21218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:51.999285 21218 hybrid_clock.cc:648] HybridClock initialized: now 1790226471999269 us; error 32 us; skew 500 ppm
Sep 24 05:07:51 dist-test-slave-m01t krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:52.001596 21218 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:52.002034 21218 webserver.cc:460] Webserver started at http://127.0.0.1:40805/ using document root <none> and password file <none>
I20260924 05:07:52.002238 21218 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:52.002285 21218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:52.002403 21218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:52.003368 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "2c423f68278e4251852eca6b1b251d35"
format_stamp: "Formatted at 2026-09-24 05:07:51 on dist-test-slave-m01t"
server_key: "0e990668e5c0271e430a1a26876da392"
server_key_iv: "47caa6cae74f72fd0e0e662291c1fd02"
server_key_version: "encryptionkey@0"
I20260924 05:07:52.003688 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c423f68278e4251852eca6b1b251d35"
format_stamp: "Formatted at 2026-09-24 05:07:51 on dist-test-slave-m01t"
server_key: "0e990668e5c0271e430a1a26876da392"
server_key_iv: "47caa6cae74f72fd0e0e662291c1fd02"
server_key_version: "encryptionkey@0"
I20260924 05:07:52.004825 21218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:52.005458 21233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:52.005616 21218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:52.005684 21218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "2c423f68278e4251852eca6b1b251d35"
format_stamp: "Formatted at 2026-09-24 05:07:51 on dist-test-slave-m01t"
server_key: "0e990668e5c0271e430a1a26876da392"
server_key_iv: "47caa6cae74f72fd0e0e662291c1fd02"
server_key_version: "encryptionkey@0"
I20260924 05:07:52.005759 21218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:52.018150 21218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:52.020113 21218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:52.020219 21218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:52.023870 21218 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41147
I20260924 05:07:52.023900 21285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41147 every 8 connection(s)
I20260924 05:07:52.024215 21218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:52.025005 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:52.025704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21218
I20260924 05:07:52.025846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:52.026037 392 external_mini_cluster.cc:1428] Setting key 24b32c42cfea0d346920300cad4789b8
I20260924 05:07:52.027972 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35: Bootstrap starting.
I20260924 05:07:52.028544 21286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:52.028815 21286 log.cc:824] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:52.029531 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35: No bootstrap required, opened a new log
I20260924 05:07:52.031827 21286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c423f68278e4251852eca6b1b251d35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41147 } }
I20260924 05:07:52.031973 21286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:52.032017 21286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c423f68278e4251852eca6b1b251d35, State: Initialized, Role: FOLLOWER
I20260924 05:07:52.032150 21286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c423f68278e4251852eca6b1b251d35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41147 } }
I20260924 05:07:52.032228 21286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:52.032266 21286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 24 05:07:52 dist-test-slave-m01t krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:07:52.032312 21286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:52.032917 21286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c423f68278e4251852eca6b1b251d35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41147 } }
I20260924 05:07:52.033041 21286 leader_election.cc:302] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c423f68278e4251852eca6b1b251d35; no voters:
I20260924 05:07:52.033216 21286 leader_election.cc:288] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:52.033248 21291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:52.033420 21291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [term 1 LEADER]: Becoming Leader. State: Replica: 2c423f68278e4251852eca6b1b251d35, State: Running, Role: LEADER
I20260924 05:07:52.033500 21286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:52.033546 21291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c423f68278e4251852eca6b1b251d35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41147 } }
I20260924 05:07:52.033947 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c423f68278e4251852eca6b1b251d35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c423f68278e4251852eca6b1b251d35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41147 } } }
I20260924 05:07:52.034021 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:52.034206 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c423f68278e4251852eca6b1b251d35. Latest consensus state: current_term: 1 leader_uuid: "2c423f68278e4251852eca6b1b251d35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c423f68278e4251852eca6b1b251d35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41147 } } }
I20260924 05:07:52.034471 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:52.034652 21297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:52.035171 21297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:52.036535 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:52.026539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:58687 (local address 127.0.0.1:41147)
0924 05:07:52.026690 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:52.026698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:52.026723 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:07:52.027130 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:52.027137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:52.027187 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:52.027296 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:52.028017 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:52.028992 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:52.029847 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:52.030445 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:52.030568 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:52.032779 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:52.032784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:52.032784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:52.032795 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:52.034908 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:52.035399 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:52.035403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:52.035403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:52.035510 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:52.035731 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:52.035732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:52.035732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:52.035966 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:52.036121 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:52.036245 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 05:07:52.036342 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 05:07:52.037350 21297 catalog_manager.cc:1347] Generated new cluster ID: 0b0d4e2f86164f5a8b315c11e42aea59
I20260924 05:07:52.037412 21297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:52.037520 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:52.044107 21297 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:52.044629 21297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:52.048857 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:52.038987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:50057 (local address 127.0.0.1:41147)
0924 05:07:52.039037 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:52.039039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:52.041993 (+ 2954us) server_negotiation.cc:407] Connection header received
0924 05:07:52.042064 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:52.042065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:52.042075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:52.042121 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:52.042724 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:52.043450 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:52.044471 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:52.044763 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:52.044828 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:52.046208 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:52.046210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:52.046210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:52.046216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:52.047651 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:52.048057 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:52.048058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:52.048058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:52.048106 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:52.048301 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:52.048302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:52.048302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:52.048404 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:52.048497 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:52.048629 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 05:07:52.048683 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:07:52.051390 21297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c423f68278e4251852eca6b1b251d35: Generated new TSK 0
I20260924 05:07:52.051529 21297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:52.053912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21218
W20260924 05:07:52.057665 21203 connection.cc:535] client connection to 127.0.0.1:41147 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41147): Connection reset by peer (error 104)
2026-09-24T05:07:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:52 dist-test-slave-m01t krb5kdc[21317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:52 dist-test-slave-m01t krb5kdc[21317](info): set up 2 sockets
Sep 24 05:07:52 dist-test-slave-m01t krb5kdc[21317](info): commencing operation
krb5kdc: starting...
W20260924 05:07:52.997517 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:53 dist-test-slave-m01t krb5kdc[21317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:53.111711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:53.190410 21330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:53.190615 21330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:53.190636 21330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:53.191723 21330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:53.191757 21330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:53.191771 21330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:53.191780 21330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:53.191793 21330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:53.193171 21330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:53.193358 21330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:53.193526 21330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:53.198066 21338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:53.198060 21335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:53.198060 21336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:53.198076 21330 server_base.cc:1034] running on GCE node
I20260924 05:07:53.198515 21330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:53.198777 21330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:53.199921 21330 hybrid_clock.cc:648] HybridClock initialized: now 1790226473199905 us; error 28 us; skew 500 ppm
Sep 24 05:07:53 dist-test-slave-m01t krb5kdc[21317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:53.202205 21330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:53.202623 21330 webserver.cc:460] Webserver started at http://127.0.0.1:36367/ using document root <none> and password file <none>
I20260924 05:07:53.202855 21330 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:53.202920 21330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:53.203047 21330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:53.204075 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "6e19045c026a404d86b67f2ad8895595"
format_stamp: "Formatted at 2026-09-24 05:07:53 on dist-test-slave-m01t"
server_key: "d52963dc219fd790f4188d8ff87422a2"
server_key_iv: "27af98576c6e154405442b81f0f1ea07"
server_key_version: "encryptionkey@0"
I20260924 05:07:53.204396 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e19045c026a404d86b67f2ad8895595"
format_stamp: "Formatted at 2026-09-24 05:07:53 on dist-test-slave-m01t"
server_key: "d52963dc219fd790f4188d8ff87422a2"
server_key_iv: "27af98576c6e154405442b81f0f1ea07"
server_key_version: "encryptionkey@0"
I20260924 05:07:53.205569 21330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:53.206246 21345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:53.206403 21330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:53.206481 21330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "6e19045c026a404d86b67f2ad8895595"
format_stamp: "Formatted at 2026-09-24 05:07:53 on dist-test-slave-m01t"
server_key: "d52963dc219fd790f4188d8ff87422a2"
server_key_iv: "27af98576c6e154405442b81f0f1ea07"
server_key_version: "encryptionkey@0"
I20260924 05:07:53.206543 21330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:53.213948 21330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:53.215574 21330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:53.215795 21330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:53.219763 21330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41897
I20260924 05:07:53.219911 21397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41897 every 8 connection(s)
I20260924 05:07:53.220191 21330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:53.221050 21398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:53.223315 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595: Bootstrap starting.
I20260924 05:07:53.223865 21398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:53.224123 21398 log.cc:824] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:53.224869 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595: No bootstrap required, opened a new log
I20260924 05:07:53.225519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21330
I20260924 05:07:53.225648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:53.225858 392 external_mini_cluster.cc:1428] Setting key ff0349f60bb5fdbade32a7a5d25e0888
I20260924 05:07:53.226927 21398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e19045c026a404d86b67f2ad8895595" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41897 } }
I20260924 05:07:53.227082 21398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:53.227161 21398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e19045c026a404d86b67f2ad8895595, State: Initialized, Role: FOLLOWER
I20260924 05:07:53.227324 21398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e19045c026a404d86b67f2ad8895595" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41897 } }
I20260924 05:07:53.227401 21398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:53.227432 21398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:53.227481 21398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:53.228021 21398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e19045c026a404d86b67f2ad8895595" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41897 } }
I20260924 05:07:53.228124 21398 leader_election.cc:302] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e19045c026a404d86b67f2ad8895595; no voters:
I20260924 05:07:53.228276 21398 leader_election.cc:288] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:53.228307 21403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:53.228402 21403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [term 1 LEADER]: Becoming Leader. State: Replica: 6e19045c026a404d86b67f2ad8895595, State: Running, Role: LEADER
I20260924 05:07:53.228535 21398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:53.228533 21403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e19045c026a404d86b67f2ad8895595" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41897 } }
I20260924 05:07:53.228906 21403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e19045c026a404d86b67f2ad8895595. Latest consensus state: current_term: 1 leader_uuid: "6e19045c026a404d86b67f2ad8895595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e19045c026a404d86b67f2ad8895595" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41897 } } }
I20260924 05:07:53.229034 21403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:53.229292 21411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:53.229848 21411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:53.230871 21404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e19045c026a404d86b67f2ad8895595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e19045c026a404d86b67f2ad8895595" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41897 } } }
I20260924 05:07:53.230978 21404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:53.231839 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:53.226392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:45261 (local address 127.0.0.1:41897)
0924 05:07:53.226583 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:53.226591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:53.226616 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:07:53.226893 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:53.226899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:53.226948 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:53.227071 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:53.227316 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:53.228681 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:53.229775 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:53.230363 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:53.230475 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:53.231233 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:53.231237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:53.231237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:53.231406 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:53.231439 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:53.231450 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:53.231521 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:53.231623 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 05:07:53.231652 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260924 05:07:53.231978 21411 catalog_manager.cc:1347] Generated new cluster ID: 5bba4d8a223e4591abd2ba7db5c30a5d
I20260924 05:07:53.232026 21411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:53.232693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:53.240516 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:53.233982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:50629 (local address 127.0.0.1:41897)
0924 05:07:53.234015 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:53.234016 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:53.238010 (+ 3994us) server_negotiation.cc:407] Connection header received
0924 05:07:53.238050 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:53.238050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:53.238063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:53.238107 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:53.238225 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:53.238706 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:53.239381 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:53.239599 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:53.239654 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:53.240285 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:53.240286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:53.240286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:53.240291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:53.240305 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:53.240310 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:53.240358 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:53.240424 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:07:53.240434 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:07:53.243590 21411 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:53.244305 21411 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:53.259690 21411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e19045c026a404d86b67f2ad8895595: Generated new TSK 0
I20260924 05:07:53.260007 21411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:53.265451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21330
W20260924 05:07:53.269258 21315 connection.cc:535] client connection to 127.0.0.1:41897 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41897): Connection reset by peer (error 104)
2026-09-24T05:07:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:53 dist-test-slave-m01t krb5kdc[21429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:53 dist-test-slave-m01t krb5kdc[21429](info): set up 2 sockets
Sep 24 05:07:53 dist-test-slave-m01t krb5kdc[21429](info): commencing operation
krb5kdc: starting...
W20260924 05:07:54.217410 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:54 dist-test-slave-m01t krb5kdc[21429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:54.318365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:54.395678 21442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:54.395867 21442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:54.395887 21442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:54.396974 21442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:54.397012 21442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:54.397025 21442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:54.397048 21442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:54.397071 21442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:54.398500 21442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:54.398717 21442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:54.398903 21442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:54.403198 21450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:54.403343 21442 server_base.cc:1034] running on GCE node
W20260924 05:07:54.403210 21447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:54.403183 21448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:54.403678 21442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:54.403905 21442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:54.405026 21442 hybrid_clock.cc:648] HybridClock initialized: now 1790226474405014 us; error 27 us; skew 500 ppm
Sep 24 05:07:54 dist-test-slave-m01t krb5kdc[21429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:54.407207 21442 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:54.407567 21442 webserver.cc:460] Webserver started at http://127.0.0.1:42083/ using document root <none> and password file <none>
I20260924 05:07:54.407760 21442 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:54.407804 21442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:54.407922 21442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:54.408869 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "6321636ca20c4ff988a6f6d69c5ce2f2"
format_stamp: "Formatted at 2026-09-24 05:07:54 on dist-test-slave-m01t"
server_key: "37f046077479f1bf1b60584351ea498c"
server_key_iv: "58b2cb65bc247b9441ac9a32ab5a0b43"
server_key_version: "encryptionkey@0"
I20260924 05:07:54.409183 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "6321636ca20c4ff988a6f6d69c5ce2f2"
format_stamp: "Formatted at 2026-09-24 05:07:54 on dist-test-slave-m01t"
server_key: "37f046077479f1bf1b60584351ea498c"
server_key_iv: "58b2cb65bc247b9441ac9a32ab5a0b43"
server_key_version: "encryptionkey@0"
I20260924 05:07:54.410377 21442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:07:54.410956 21457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:54.411108 21442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:54.411180 21442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "6321636ca20c4ff988a6f6d69c5ce2f2"
format_stamp: "Formatted at 2026-09-24 05:07:54 on dist-test-slave-m01t"
server_key: "37f046077479f1bf1b60584351ea498c"
server_key_iv: "58b2cb65bc247b9441ac9a32ab5a0b43"
server_key_version: "encryptionkey@0"
I20260924 05:07:54.411237 21442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:54.414248 21442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:54.416132 21442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:54.416262 21442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:54.419760 21442 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34027
I20260924 05:07:54.419788 21509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34027 every 8 connection(s)
I20260924 05:07:54.420073 21442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:54.420950 21510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:54.421972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21442
I20260924 05:07:54.422107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:54.422324 392 external_mini_cluster.cc:1428] Setting key 1dda6c2d5e53db95314a72697bc063a6
I20260924 05:07:54.423473 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2: Bootstrap starting.
I20260924 05:07:54.424114 21510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:54.424407 21510 log.cc:824] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:54.424654 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:54.422828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:35791 (local address 127.0.0.1:34027)
0924 05:07:54.423000 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:54.423008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:54.423032 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:07:54.423434 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:54.423440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:54.423466 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:54.423606 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:54.424226 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:54.424230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:54.424230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:54.424235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:54.424285 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:54.424298 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:54.424378 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:54.424435 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 05:07:54.424474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260924 05:07:54.425153 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2: No bootstrap required, opened a new log
I20260924 05:07:54.425626 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:54.427493 21510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260924 05:07:54.427639 21510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:54.427670 21510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6321636ca20c4ff988a6f6d69c5ce2f2, State: Initialized, Role: FOLLOWER
I20260924 05:07:54.427788 21510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260924 05:07:54.427866 21510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:54.427896 21510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:54.427942 21510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:54.428318 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:54.427307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:44355 (local address 127.0.0.1:34027)
0924 05:07:54.427356 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:54.427358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:54.427471 (+ 113us) server_negotiation.cc:407] Connection header received
0924 05:07:54.427526 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:54.427527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:54.427532 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:54.427575 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:54.428131 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:54.428132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:54.428132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:54.428136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:54.428150 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:54.428154 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:54.428197 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:54.428244 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:07:54.428257 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:07:54.428478 21510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260924 05:07:54.428614 21510 leader_election.cc:302] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6321636ca20c4ff988a6f6d69c5ce2f2; no voters:
I20260924 05:07:54.428771 21510 leader_election.cc:288] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:54.428795 21520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:54.428913 21520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6321636ca20c4ff988a6f6d69c5ce2f2, State: Running, Role: LEADER
I20260924 05:07:54.429013 21510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:54.429057 21520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } }
I20260924 05:07:54.429443 21521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } } }
I20260924 05:07:54.429572 21521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:54.429826 21529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:54.429443 21522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6321636ca20c4ff988a6f6d69c5ce2f2. Latest consensus state: current_term: 1 leader_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6321636ca20c4ff988a6f6d69c5ce2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34027 } } }
I20260924 05:07:54.430184 21522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:54.430287 21529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:54.431643 21529 catalog_manager.cc:1347] Generated new cluster ID: 9f1e2c54a30442c9956e9ef7f274c6d6
I20260924 05:07:54.431691 21529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:54.440590 21529 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:54.440732 21529 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:54.443166 21529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6321636ca20c4ff988a6f6d69c5ce2f2: Generated new TSK 0
I20260924 05:07:54.443306 21529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:54.448628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21442
2026-09-24T05:07:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:54 dist-test-slave-m01t krb5kdc[21541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:54 dist-test-slave-m01t krb5kdc[21541](info): set up 2 sockets
Sep 24 05:07:54 dist-test-slave-m01t krb5kdc[21541](info): commencing operation
krb5kdc: starting...
W20260924 05:07:55.377291 392 mini_kdc.cc:121] Time spent starting KDC: real 0.912s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:55 dist-test-slave-m01t krb5kdc[21541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:55.482374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:55.562552 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:55.562739 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:55.562768 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:55.563865 21554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:55.563906 21554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:55.563923 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:55.563939 21554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:55.563953 21554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:55.565361 21554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:55.565596 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:55.565804 21554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:55.570139 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:55.570219 21554 server_base.cc:1034] running on GCE node
W20260924 05:07:55.570148 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:55.570202 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:55.570503 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:55.570770 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:55.571897 21554 hybrid_clock.cc:648] HybridClock initialized: now 1790226475571889 us; error 25 us; skew 500 ppm
Sep 24 05:07:55 dist-test-slave-m01t krb5kdc[21541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:55.574252 21554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:55.574613 21554 webserver.cc:460] Webserver started at http://127.0.0.1:44175/ using document root <none> and password file <none>
I20260924 05:07:55.574790 21554 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:55.574828 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:55.574913 21554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:55.575839 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "d6738351d6b54f2bb0124e53dadac95d"
format_stamp: "Formatted at 2026-09-24 05:07:55 on dist-test-slave-m01t"
server_key: "ef9188d85f0641f5f7bc1ed1d2fea1d2"
server_key_iv: "c3dfb17434f56107624b4b094601bff0"
server_key_version: "encryptionkey@0"
I20260924 05:07:55.576112 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6738351d6b54f2bb0124e53dadac95d"
format_stamp: "Formatted at 2026-09-24 05:07:55 on dist-test-slave-m01t"
server_key: "ef9188d85f0641f5f7bc1ed1d2fea1d2"
server_key_iv: "c3dfb17434f56107624b4b094601bff0"
server_key_version: "encryptionkey@0"
I20260924 05:07:55.577241 21554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:07:55.577905 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:55.578049 21554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:55.578125 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "d6738351d6b54f2bb0124e53dadac95d"
format_stamp: "Formatted at 2026-09-24 05:07:55 on dist-test-slave-m01t"
server_key: "ef9188d85f0641f5f7bc1ed1d2fea1d2"
server_key_iv: "c3dfb17434f56107624b4b094601bff0"
server_key_version: "encryptionkey@0"
I20260924 05:07:55.578181 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:55.585780 21554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:55.587678 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:55.587779 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:55.591252 21554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33317
I20260924 05:07:55.591295 21621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33317 every 8 connection(s)
I20260924 05:07:55.591581 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:55.592353 21622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:55.594589 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d: Bootstrap starting.
I20260924 05:07:55.595121 21622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:55.595364 21622 log.cc:824] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:55.595733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21554
I20260924 05:07:55.595846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:55.596025 392 external_mini_cluster.cc:1428] Setting key c5bba2f2752c6bdfdd9634fbf8d48bf8
I20260924 05:07:55.596061 21622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d: No bootstrap required, opened a new log
I20260924 05:07:55.598085 21622 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6738351d6b54f2bb0124e53dadac95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260924 05:07:55.598253 21622 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:55.598284 21622 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6738351d6b54f2bb0124e53dadac95d, State: Initialized, Role: FOLLOWER
I20260924 05:07:55.598413 21622 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6738351d6b54f2bb0124e53dadac95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260924 05:07:55.598489 21622 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:55.598526 21622 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:55.598577 21622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:55.599215 21622 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6738351d6b54f2bb0124e53dadac95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260924 05:07:55.599321 21622 leader_election.cc:302] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6738351d6b54f2bb0124e53dadac95d; no voters:
I20260924 05:07:55.599464 21622 leader_election.cc:288] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:55.599498 21627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:55.599640 21627 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [term 1 LEADER]: Becoming Leader. State: Replica: d6738351d6b54f2bb0124e53dadac95d, State: Running, Role: LEADER
I20260924 05:07:55.599684 21622 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:55.600768 21627 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6738351d6b54f2bb0124e53dadac95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260924 05:07:55.601307 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6738351d6b54f2bb0124e53dadac95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6738351d6b54f2bb0124e53dadac95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } } }
I20260924 05:07:55.601385 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:55.601599 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6738351d6b54f2bb0124e53dadac95d. Latest consensus state: current_term: 1 leader_uuid: "d6738351d6b54f2bb0124e53dadac95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6738351d6b54f2bb0124e53dadac95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } } }
I20260924 05:07:55.601673 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:55.601729 21641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:55.602253 21641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:07:55 dist-test-slave-m01t krb5kdc[21541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:07:55.603690 21641 catalog_manager.cc:1347] Generated new cluster ID: 12270d89e31643de86658c82e44ea28a
I20260924 05:07:55.603739 21641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:55.607609 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:55.596491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:34841 (local address 127.0.0.1:33317)
0924 05:07:55.596662 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:55.596672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:55.596701 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:07:55.597267 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:55.597277 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:55.597342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:55.597501 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:55.598096 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:55.599705 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:55.600643 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:55.601229 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:55.601336 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:55.604034 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:55.604037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:55.604038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:55.604048 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:55.605816 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:55.606230 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:55.606233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:55.606233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:55.606323 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:55.606526 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:55.606526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:55.606527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:55.606803 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:55.607245 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:55.607326 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 05:07:55.607415 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260924 05:07:55.608443 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:55.611205 21641 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:55.611814 21641 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:55.616389 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:55.609814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:45901 (local address 127.0.0.1:33317)
0924 05:07:55.609847 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:55.609849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:55.610003 (+ 154us) server_negotiation.cc:407] Connection header received
0924 05:07:55.610084 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:55.610084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:55.610092 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:55.610117 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:55.610729 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:55.611355 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:55.612297 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:55.612611 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:55.612650 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:55.613950 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:55.613951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:55.613951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:55.613957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:55.615287 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:55.615667 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:55.615668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:55.615668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:55.615714 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:55.615894 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:55.615894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:55.615895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:55.615994 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:55.616057 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:55.616254 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 05:07:55.616308 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:07:55.617785 21641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6738351d6b54f2bb0124e53dadac95d: Generated new TSK 0
I20260924 05:07:55.617941 21641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:55.622211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21554
W20260924 05:07:55.626085 21539 connection.cc:535] client connection to 127.0.0.1:33317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33317): Connection reset by peer (error 104)
2026-09-24T05:07:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:55 dist-test-slave-m01t krb5kdc[21653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:55 dist-test-slave-m01t krb5kdc[21653](info): set up 2 sockets
Sep 24 05:07:55 dist-test-slave-m01t krb5kdc[21653](info): commencing operation
krb5kdc: starting...
W20260924 05:07:56.565506 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:56 dist-test-slave-m01t krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:56.670037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:56.746969 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:56.747165 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:56.747195 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:56.748317 21666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:56.748360 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:56.748377 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:56.748392 21666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:56.748407 21666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:56.749876 21666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:56.750108 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:56.750305 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:56.754623 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:56.754690 21666 server_base.cc:1034] running on GCE node
W20260924 05:07:56.754575 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:56.754585 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:56.755038 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:56.755303 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:56.756428 21666 hybrid_clock.cc:648] HybridClock initialized: now 1790226476756409 us; error 38 us; skew 500 ppm
Sep 24 05:07:56 dist-test-slave-m01t krb5kdc[21653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:56.763513 21666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:56.763924 21666 webserver.cc:460] Webserver started at http://127.0.0.1:37943/ using document root <none> and password file <none>
I20260924 05:07:56.764113 21666 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:56.764155 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:56.764273 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:56.765237 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "12d1947b290b4a18802e9fe0bf8fd70c"
format_stamp: "Formatted at 2026-09-24 05:07:56 on dist-test-slave-m01t"
server_key: "3ce76b93b6972a5c6568290dcd1816e6"
server_key_iv: "db393e9ad67d6691d4f74c4a87114b5d"
server_key_version: "encryptionkey@0"
I20260924 05:07:56.765549 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "12d1947b290b4a18802e9fe0bf8fd70c"
format_stamp: "Formatted at 2026-09-24 05:07:56 on dist-test-slave-m01t"
server_key: "3ce76b93b6972a5c6568290dcd1816e6"
server_key_iv: "db393e9ad67d6691d4f74c4a87114b5d"
server_key_version: "encryptionkey@0"
I20260924 05:07:56.766727 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:07:56.767344 21681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:56.767473 21666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:56.767557 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "12d1947b290b4a18802e9fe0bf8fd70c"
format_stamp: "Formatted at 2026-09-24 05:07:56 on dist-test-slave-m01t"
server_key: "3ce76b93b6972a5c6568290dcd1816e6"
server_key_iv: "db393e9ad67d6691d4f74c4a87114b5d"
server_key_version: "encryptionkey@0"
I20260924 05:07:56.767614 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:56.789891 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:56.791682 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:56.791817 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:56.795650 21666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40237
I20260924 05:07:56.795692 21733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40237 every 8 connection(s)
I20260924 05:07:56.796016 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:56.796890 21734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:56.799060 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c: Bootstrap starting.
I20260924 05:07:56.799546 21734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:56.799763 21734 log.cc:824] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:56.800349 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c: No bootstrap required, opened a new log
I20260924 05:07:56.801857 21734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260924 05:07:56.801994 21734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:56.802016 21734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d1947b290b4a18802e9fe0bf8fd70c, State: Initialized, Role: FOLLOWER
I20260924 05:07:56.802115 21734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260924 05:07:56.802160 21734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:56.802183 21734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:56.802212 21734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:56.802646 21734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260924 05:07:56.802727 21734 leader_election.cc:302] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12d1947b290b4a18802e9fe0bf8fd70c; no voters:
I20260924 05:07:56.802835 21734 leader_election.cc:288] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:56.802896 21737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:56.803040 21734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:56.803047 21737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [term 1 LEADER]: Becoming Leader. State: Replica: 12d1947b290b4a18802e9fe0bf8fd70c, State: Running, Role: LEADER
I20260924 05:07:56.803146 21737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260924 05:07:56.803623 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d1947b290b4a18802e9fe0bf8fd70c. Latest consensus state: current_term: 1 leader_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260924 05:07:56.803705 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:56.803774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21666
I20260924 05:07:56.803876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:56.804059 392 external_mini_cluster.cc:1428] Setting key 16cd41b99cbd00764f420327e7323ccc
I20260924 05:07:56.804072 21746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:56.804054 21738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d1947b290b4a18802e9fe0bf8fd70c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260924 05:07:56.804147 21738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:56.804570 21746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:56.805996 21746 catalog_manager.cc:1347] Generated new cluster ID: 419a715829274b8ca2806fcfad3c3e79
I20260924 05:07:56.806044 21746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:56.809926 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:56.804520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:48029 (local address 127.0.0.1:40237)
0924 05:07:56.804744 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:56.804754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:56.804787 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:07:56.804901 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:56.804907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:56.804982 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:56.805115 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:56.805362 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:56.806719 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:56.807761 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:56.808399 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:56.808510 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:56.809143 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:56.809148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:56.809149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:56.809343 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:56.809384 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:56.809400 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:56.809486 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:56.809646 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 05:07:56.809678 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260924 05:07:56.810765 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:56.815488 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:56.812249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:55825 (local address 127.0.0.1:40237)
0924 05:07:56.812281 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:56.812282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:56.812351 (+ 69us) server_negotiation.cc:407] Connection header received
0924 05:07:56.812401 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:56.812401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:56.812411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:56.812476 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:56.812578 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:56.813298 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:56.814259 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:56.814564 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:56.814638 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:56.815257 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:56.815258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:56.815258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:56.815263 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:56.815277 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:56.815282 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:56.815341 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:56.815401 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:07:56.815413 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:07:56.816354 21746 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:56.816807 21746 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:56.823732 21746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12d1947b290b4a18802e9fe0bf8fd70c: Generated new TSK 0
I20260924 05:07:56.823907 21746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:56.830272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21666
W20260924 05:07:56.834131 21651 connection.cc:535] client connection to 127.0.0.1:40237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40237): Connection reset by peer (error 104)
2026-09-24T05:07:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:56 dist-test-slave-m01t krb5kdc[21765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:56 dist-test-slave-m01t krb5kdc[21765](info): set up 2 sockets
Sep 24 05:07:56 dist-test-slave-m01t krb5kdc[21765](info): commencing operation
krb5kdc: starting...
W20260924 05:07:57.773356 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:57 dist-test-slave-m01t krb5kdc[21765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:07:57.880028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:57.957355 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:57.957548 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:57.957578 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:57.958720 21778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:57.958762 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:57.958781 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:57.958796 21778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:57.958811 21778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:57.960235 21778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:57.960464 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:57.960670 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:57.965050 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:57.965027 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:57.965111 21778 server_base.cc:1034] running on GCE node
W20260924 05:07:57.965029 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:57.965453 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:57.965740 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:57.966878 21778 hybrid_clock.cc:648] HybridClock initialized: now 1790226477966869 us; error 27 us; skew 500 ppm
Sep 24 05:07:57 dist-test-slave-m01t krb5kdc[21765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:57.969009 21778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:07:57.969408 21778 webserver.cc:460] Webserver started at http://127.0.0.1:37741/ using document root <none> and password file <none>
I20260924 05:07:57.969623 21778 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:57.969671 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:57.969817 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:57.970769 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "150d35d0f83f4df48db3e949bd7f0bf9"
format_stamp: "Formatted at 2026-09-24 05:07:57 on dist-test-slave-m01t"
server_key: "550d583ae2f85f62b60fff0f2eb08d58"
server_key_iv: "ce2090e3c21e4bd9d270c6c577f6cd96"
server_key_version: "encryptionkey@0"
I20260924 05:07:57.971076 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "150d35d0f83f4df48db3e949bd7f0bf9"
format_stamp: "Formatted at 2026-09-24 05:07:57 on dist-test-slave-m01t"
server_key: "550d583ae2f85f62b60fff0f2eb08d58"
server_key_iv: "ce2090e3c21e4bd9d270c6c577f6cd96"
server_key_version: "encryptionkey@0"
I20260924 05:07:57.972275 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:07:57.972903 21793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:57.973047 21778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:07:57.973125 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "150d35d0f83f4df48db3e949bd7f0bf9"
format_stamp: "Formatted at 2026-09-24 05:07:57 on dist-test-slave-m01t"
server_key: "550d583ae2f85f62b60fff0f2eb08d58"
server_key_iv: "ce2090e3c21e4bd9d270c6c577f6cd96"
server_key_version: "encryptionkey@0"
I20260924 05:07:57.973183 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:57.976276 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:57.977924 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:57.978041 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:57.981549 21778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33319
I20260924 05:07:57.981568 21845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33319 every 8 connection(s)
I20260924 05:07:57.981878 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:57.982720 21846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:57.983670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21778
I20260924 05:07:57.983772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:57.983942 392 external_mini_cluster.cc:1428] Setting key 7f277210c8d275489c25d525049aa772
I20260924 05:07:57.985009 21846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9: Bootstrap starting.
I20260924 05:07:57.985615 21846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:57.985958 21846 log.cc:824] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:57.986143 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:57.984385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:45259 (local address 127.0.0.1:33319)
0924 05:07:57.984537 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:57.984544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:57.984568 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:07:57.984957 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:57.984962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:57.984985 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:57.985132 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:57.985748 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:57.985751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:57.985751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:57.985755 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:57.985793 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:57.985802 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:57.985868 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:57.985931 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 05:07:57.985966 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 05:07:57.986595 21846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9: No bootstrap required, opened a new log
I20260924 05:07:57.987038 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:57.988533 21846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260924 05:07:57.988667 21846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:57.988700 21846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150d35d0f83f4df48db3e949bd7f0bf9, State: Initialized, Role: FOLLOWER
I20260924 05:07:57.988809 21846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260924 05:07:57.988900 21846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:57.988934 21846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:57.988982 21846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:57.989207 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:57.988247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:34387 (local address 127.0.0.1:33319)
0924 05:07:57.988277 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:57.988279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:57.988400 (+ 121us) server_negotiation.cc:407] Connection header received
0924 05:07:57.988465 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:57.988466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:57.988470 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:57.988513 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:57.989027 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:57.989029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:57.989029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:07:57.989033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:57.989044 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:07:57.989048 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:57.989091 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:57.989138 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:07:57.989150 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:07:57.989593 21846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260924 05:07:57.989698 21846 leader_election.cc:302] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 150d35d0f83f4df48db3e949bd7f0bf9; no voters:
I20260924 05:07:57.989876 21846 leader_election.cc:288] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:57.989903 21856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:57.990036 21856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 150d35d0f83f4df48db3e949bd7f0bf9, State: Running, Role: LEADER
I20260924 05:07:57.990097 21846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:57.990141 21856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260924 05:07:57.990517 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 150d35d0f83f4df48db3e949bd7f0bf9. Latest consensus state: current_term: 1 leader_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } } }
I20260924 05:07:57.990618 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:57.990795 21865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:57.990973 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150d35d0f83f4df48db3e949bd7f0bf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } } }
I20260924 05:07:57.991123 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:57.991228 21865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:57.992574 21865 catalog_manager.cc:1347] Generated new cluster ID: 91a46db27859443f814717e2b97cb28e
I20260924 05:07:57.992640 21865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:07:57.996156 21865 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:57.996291 21865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:58.001158 21865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 150d35d0f83f4df48db3e949bd7f0bf9: Generated new TSK 0
I20260924 05:07:58.001295 21865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:58.009517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21778
2026-09-24T05:07:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1179 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:58 dist-test-slave-m01t krb5kdc[21877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:58 dist-test-slave-m01t krb5kdc[21877](info): set up 2 sockets
Sep 24 05:07:58 dist-test-slave-m01t krb5kdc[21877](info): commencing operation
krb5kdc: starting...
W20260924 05:07:58.957484 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:07:59 dist-test-slave-m01t krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:07:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:07:59.062222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45073
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:07:59.140594 21890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:07:59.140805 21890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:07:59.140836 21890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:07:59.142035 21890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:07:59.142076 21890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:07:59.142096 21890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:07:59.142112 21890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:07:59.142128 21890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:07:59.143569 21890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:07:59.143808 21890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:07:59.144008 21890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:07:59.148412 21896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:59.148433 21895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:07:59.148432 21898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:07:59.148643 21890 server_base.cc:1034] running on GCE node
I20260924 05:07:59.148913 21890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:07:59.149168 21890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:07:59.150300 21890 hybrid_clock.cc:648] HybridClock initialized: now 1790226479150287 us; error 29 us; skew 500 ppm
Sep 24 05:07:59 dist-test-slave-m01t krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:07:59.152565 21890 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:07:59.152948 21890 webserver.cc:460] Webserver started at http://127.0.98.62:32811/ using document root <none> and password file <none>
I20260924 05:07:59.153133 21890 fs_manager.cc:359] Metadata directory not provided
I20260924 05:07:59.153179 21890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:07:59.153301 21890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:07:59.154330 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "8ff5e98474164412ad1e0d54f3665d88"
format_stamp: "Formatted at 2026-09-24 05:07:59 on dist-test-slave-m01t"
server_key: "fd7d9deece395fd0e3df36a8060a52f7"
server_key_iv: "f837a3949d3584fdd66a07c76b6e0c60"
server_key_version: "encryptionkey@0"
I20260924 05:07:59.154657 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ff5e98474164412ad1e0d54f3665d88"
format_stamp: "Formatted at 2026-09-24 05:07:59 on dist-test-slave-m01t"
server_key: "fd7d9deece395fd0e3df36a8060a52f7"
server_key_iv: "f837a3949d3584fdd66a07c76b6e0c60"
server_key_version: "encryptionkey@0"
I20260924 05:07:59.155833 21890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:07:59.156437 21905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:07:59.156582 21890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:07:59.156661 21890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "8ff5e98474164412ad1e0d54f3665d88"
format_stamp: "Formatted at 2026-09-24 05:07:59 on dist-test-slave-m01t"
server_key: "fd7d9deece395fd0e3df36a8060a52f7"
server_key_iv: "f837a3949d3584fdd66a07c76b6e0c60"
server_key_version: "encryptionkey@0"
I20260924 05:07:59.156718 21890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:07:59.170903 21890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:07:59.172458 21890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:07:59.172585 21890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:07:59.176079 21890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45073
I20260924 05:07:59.176131 21957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45073 every 8 connection(s)
I20260924 05:07:59.176422 21890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:07:59.177191 21958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:07:59.179463 21958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88: Bootstrap starting.
I20260924 05:07:59.179991 21958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88: Neither blocks nor log segments found. Creating new log.
I20260924 05:07:59.180235 21958 log.cc:824] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88: Log is configured to *not* fsync() on all Append() calls
I20260924 05:07:59.180837 21958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88: No bootstrap required, opened a new log
I20260924 05:07:59.182407 21958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff5e98474164412ad1e0d54f3665d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260924 05:07:59.182547 21958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:07:59.182571 21958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff5e98474164412ad1e0d54f3665d88, State: Initialized, Role: FOLLOWER
I20260924 05:07:59.182706 21958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff5e98474164412ad1e0d54f3665d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260924 05:07:59.182778 21958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:07:59.182811 21958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:07:59.182857 21958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:07:59.183337 21958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff5e98474164412ad1e0d54f3665d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260924 05:07:59.183426 21958 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff5e98474164412ad1e0d54f3665d88; no voters:
I20260924 05:07:59.183579 21958 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:07:59.183619 21961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:07:59.183763 21961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff5e98474164412ad1e0d54f3665d88, State: Running, Role: LEADER
I20260924 05:07:59.183802 21958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:07:59.183852 21961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff5e98474164412ad1e0d54f3665d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } }
I20260924 05:07:59.184232 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff5e98474164412ad1e0d54f3665d88. Latest consensus state: current_term: 1 leader_uuid: "8ff5e98474164412ad1e0d54f3665d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff5e98474164412ad1e0d54f3665d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } } }
I20260924 05:07:59.184306 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:59.184350 21962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff5e98474164412ad1e0d54f3665d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff5e98474164412ad1e0d54f3665d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45073 } } }
I20260924 05:07:59.184418 21962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88 [sys.catalog]: This master's current role is: LEADER
I20260924 05:07:59.184827 21970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:07:59.185295 21970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:07:59.186031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 21890
I20260924 05:07:59.186149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:07:59.186331 392 external_mini_cluster.cc:1428] Setting key d757b7c4e41375fac9f51c822c2078dd
I20260924 05:07:59.187211 21970 catalog_manager.cc:1347] Generated new cluster ID: 3d90e0748f8a4d53abc0bd1245766153
I20260924 05:07:59.187256 21970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:07:59 dist-test-slave-m01t krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:07:59.199867 21970 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:07:59.200251 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:59.186783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54897 (local address 127.0.98.62:45073)
0924 05:07:59.189821 (+ 3038us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:59.189829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:59.189864 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:07:59.189976 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:59.189983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:59.190042 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:59.190176 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:59.190920 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:59.192269 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:59.193314 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:59.193915 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:59.194002 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:59.196677 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:59.196681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:59.196681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:59.196693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:59.198628 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:59.199061 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:59.199064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:59.199065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:59.199172 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:59.199393 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:59.199394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:59.199395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:59.199685 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:59.199822 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:59.199961 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 05:07:59.200061 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":62,"threads_started":1}
I20260924 05:07:59.200717 21970 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:07:59.201251 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:07:59.209642 21970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff5e98474164412ad1e0d54f3665d88: Generated new TSK 0
I20260924 05:07:59.209843 21970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:07:59.211912 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:07:59.202548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46201 (local address 127.0.98.62:45073)
0924 05:07:59.202582 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:07:59.202583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:07:59.205879 (+ 3296us) server_negotiation.cc:407] Connection header received
0924 05:07:59.205936 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:07:59.205937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:07:59.205946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:07:59.205990 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:07:59.206472 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:59.207268 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:59.207972 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:07:59.208180 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:07:59.208232 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:07:59.209618 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:07:59.209619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:07:59.209619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:07:59.209624 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:07:59.210883 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:59.211267 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:59.211267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:59.211268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:59.211313 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:07:59.211506 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:07:59.211506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:07:59.211507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:07:59.211601 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:07:59.211671 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:07:59.211810 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 05:07:59.211853 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:07:59.213644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 21890
W20260924 05:07:59.217216 21875 connection.cc:535] client connection to 127.0.98.62:45073 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45073): Connection reset by peer (error 104)
2026-09-24T05:07:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:07:59 dist-test-slave-m01t krb5kdc[21989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:07:59 dist-test-slave-m01t krb5kdc[21989](info): set up 2 sockets
Sep 24 05:07:59 dist-test-slave-m01t krb5kdc[21989](info): commencing operation
krb5kdc: starting...
W20260924 05:08:00.153386 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:00 dist-test-slave-m01t krb5kdc[21989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:00.258682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:00.336005 22002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:00.336228 22002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:00.336251 22002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:00.337445 22002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:00.337487 22002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:00.337500 22002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:00.337515 22002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:00.337539 22002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:00.339025 22002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:00.339252 22002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:00.339459 22002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:00.343794 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:00.343935 22002 server_base.cc:1034] running on GCE node
W20260924 05:08:00.343809 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:00.343885 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:00.344267 22002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:00.344543 22002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:00.345682 22002 hybrid_clock.cc:648] HybridClock initialized: now 1790226480345665 us; error 34 us; skew 500 ppm
Sep 24 05:08:00 dist-test-slave-m01t krb5kdc[21989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:00.347944 22002 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:00.348297 22002 webserver.cc:460] Webserver started at http://127.0.98.62:38285/ using document root <none> and password file <none>
I20260924 05:08:00.348460 22002 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:00.348498 22002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:00.348594 22002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:00.349536 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "4cdcb702e99a41008052096e9f492803"
format_stamp: "Formatted at 2026-09-24 05:08:00 on dist-test-slave-m01t"
server_key: "7175215fc2962460b679319ac89995eb"
server_key_iv: "297a587cad8eb8677e80d82c79d7f055"
server_key_version: "encryptionkey@0"
I20260924 05:08:00.349870 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cdcb702e99a41008052096e9f492803"
format_stamp: "Formatted at 2026-09-24 05:08:00 on dist-test-slave-m01t"
server_key: "7175215fc2962460b679319ac89995eb"
server_key_iv: "297a587cad8eb8677e80d82c79d7f055"
server_key_version: "encryptionkey@0"
I20260924 05:08:00.350980 22002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:08:00.351589 22017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:00.351727 22002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:00.351804 22002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "4cdcb702e99a41008052096e9f492803"
format_stamp: "Formatted at 2026-09-24 05:08:00 on dist-test-slave-m01t"
server_key: "7175215fc2962460b679319ac89995eb"
server_key_iv: "297a587cad8eb8677e80d82c79d7f055"
server_key_version: "encryptionkey@0"
I20260924 05:08:00.351862 22002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:00.373224 22002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:00.375337 22002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:00.375478 22002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:00.379091 22002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35243
I20260924 05:08:00.379128 22069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35243 every 8 connection(s)
I20260924 05:08:00.379474 22002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:00.380411 22070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:00.382579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22002
I20260924 05:08:00.382637 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803: Bootstrap starting.
I20260924 05:08:00.382673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:00.382836 392 external_mini_cluster.cc:1428] Setting key 5b5f0b75e8bc0e4a9c531bb0e2b3bfc1
I20260924 05:08:00.383234 22070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:00.383544 22070 log.cc:824] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:00.384279 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803: No bootstrap required, opened a new log
I20260924 05:08:00.386292 22070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdcb702e99a41008052096e9f492803" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35243 } }
I20260924 05:08:00.386473 22070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:00.386526 22070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdcb702e99a41008052096e9f492803, State: Initialized, Role: FOLLOWER
I20260924 05:08:00.386657 22070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdcb702e99a41008052096e9f492803" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35243 } }
I20260924 05:08:00.386718 22070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:00.386739 22070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:00.386783 22070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:00.387427 22070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdcb702e99a41008052096e9f492803" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35243 } }
I20260924 05:08:00.387526 22070 leader_election.cc:302] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cdcb702e99a41008052096e9f492803; no voters:
I20260924 05:08:00.387699 22070 leader_election.cc:288] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:00.387740 22075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:00.387884 22075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [term 1 LEADER]: Becoming Leader. State: Replica: 4cdcb702e99a41008052096e9f492803, State: Running, Role: LEADER
I20260924 05:08:00.387966 22070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:00.388023 22075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdcb702e99a41008052096e9f492803" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35243 } }
I20260924 05:08:00.388360 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cdcb702e99a41008052096e9f492803. Latest consensus state: current_term: 1 leader_uuid: "4cdcb702e99a41008052096e9f492803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdcb702e99a41008052096e9f492803" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35243 } } }
I20260924 05:08:00.388547 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:00.388585 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:00.383286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52207 (local address 127.0.98.62:35243)
0924 05:08:00.383441 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:00.383449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:00.383473 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:08:00.383986 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:00.383994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:00.384046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:00.384160 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:00.384434 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:00.385302 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:00.386454 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:00.387015 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:00.387096 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:00.387835 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:00.387839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:00.387839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:00.388034 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:00.388094 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:00.388111 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:00.388174 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:00.388344 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 05:08:00.388373 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 05:08:00.388491 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cdcb702e99a41008052096e9f492803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cdcb702e99a41008052096e9f492803" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35243 } } }
I20260924 05:08:00.388854 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:00.389036 22085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:00.389436 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:00.389539 22085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:00.391327 22085 catalog_manager.cc:1347] Generated new cluster ID: 4e3807049897406887eb248482658cbd
I20260924 05:08:00.391391 22085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:00.399994 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:00.390758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52525 (local address 127.0.98.62:35243)
0924 05:08:00.390794 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:00.390796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:00.390998 (+ 202us) server_negotiation.cc:407] Connection header received
0924 05:08:00.391084 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:00.391085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:00.391093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:00.391119 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:00.397798 (+ 6679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:00.398272 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:00.398935 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:00.399140 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:00.399191 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:00.399786 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:00.399787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:00.399788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:00.399792 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:00.399805 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:00.399809 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:00.399851 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:00.399916 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:08:00.399924 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:00.406503 22085 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:00.407176 22085 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:00.413677 22085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cdcb702e99a41008052096e9f492803: Generated new TSK 0
I20260924 05:08:00.413879 22085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:08:00.418295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22002
W20260924 05:08:00.421887 21987 connection.cc:535] client connection to 127.0.98.62:35243 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35243): Connection reset by peer (error 104)
2026-09-24T05:08:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:00 dist-test-slave-m01t krb5kdc[22101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:00 dist-test-slave-m01t krb5kdc[22101](info): set up 2 sockets
Sep 24 05:08:00 dist-test-slave-m01t krb5kdc[22101](info): commencing operation
krb5kdc: starting...
W20260924 05:08:01.357434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:01 dist-test-slave-m01t krb5kdc[22101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:01.465870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:01.542656 22114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:01.542863 22114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:01.542892 22114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:01.544009 22114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:01.544051 22114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:01.544070 22114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:01.544085 22114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:01.544100 22114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:01.545521 22114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:01.545781 22114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:01.545991 22114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:01.550437 22119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:01.550431 22122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:01.550426 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:01.550483 22114 server_base.cc:1034] running on GCE node
I20260924 05:08:01.550802 22114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:01.551052 22114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:01.552181 22114 hybrid_clock.cc:648] HybridClock initialized: now 1790226481552171 us; error 28 us; skew 500 ppm
Sep 24 05:08:01 dist-test-slave-m01t krb5kdc[22101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:01.554406 22114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:01.554821 22114 webserver.cc:460] Webserver started at http://127.0.98.62:35229/ using document root <none> and password file <none>
I20260924 05:08:01.555006 22114 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:01.555052 22114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:01.555171 22114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:01.556136 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "77476c6004724e3cbbd2e56155707cb6"
format_stamp: "Formatted at 2026-09-24 05:08:01 on dist-test-slave-m01t"
server_key: "05d9d0ec824567b3ad36db09d419ddec"
server_key_iv: "141ceafb18230c99a211a0f7f882f7eb"
server_key_version: "encryptionkey@0"
I20260924 05:08:01.556445 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "77476c6004724e3cbbd2e56155707cb6"
format_stamp: "Formatted at 2026-09-24 05:08:01 on dist-test-slave-m01t"
server_key: "05d9d0ec824567b3ad36db09d419ddec"
server_key_iv: "141ceafb18230c99a211a0f7f882f7eb"
server_key_version: "encryptionkey@0"
I20260924 05:08:01.557583 22114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:01.558292 22129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:01.558444 22114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:01.558521 22114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "77476c6004724e3cbbd2e56155707cb6"
format_stamp: "Formatted at 2026-09-24 05:08:01 on dist-test-slave-m01t"
server_key: "05d9d0ec824567b3ad36db09d419ddec"
server_key_iv: "141ceafb18230c99a211a0f7f882f7eb"
server_key_version: "encryptionkey@0"
I20260924 05:08:01.558585 22114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:01.561821 22114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:01.563742 22114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:01.563870 22114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:01.567353 22114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39927
I20260924 05:08:01.567381 22181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39927 every 8 connection(s)
I20260924 05:08:01.567652 22114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:01.568480 22182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:01.569396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22114
I20260924 05:08:01.569523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:01.569733 392 external_mini_cluster.cc:1428] Setting key 2ff3fac6a86f4d99871cf123fe33f7c6
I20260924 05:08:01.571072 22182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6: Bootstrap starting.
I20260924 05:08:01.571715 22182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:01.571923 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:01.570201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47963 (local address 127.0.98.62:39927)
0924 05:08:01.570373 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:01.570382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:01.570412 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:08:01.570873 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:01.570882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:01.570915 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:01.571077 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:01.571540 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:01.571543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:01.571543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:01.571548 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:01.571585 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:01.571599 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:01.571684 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:01.571715 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 05:08:01.571753 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260924 05:08:01.572009 22182 log.cc:824] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:01.572660 22182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6: No bootstrap required, opened a new log
I20260924 05:08:01.572783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:01.574994 22182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77476c6004724e3cbbd2e56155707cb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260924 05:08:01.575119 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:01.574242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36541 (local address 127.0.98.62:39927)
0924 05:08:01.574278 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:01.574280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:01.574405 (+ 125us) server_negotiation.cc:407] Connection header received
0924 05:08:01.574466 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:01.574467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:01.574471 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:01.574507 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:01.574953 (+ 446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:01.574954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:01.574954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:01.574958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:01.574968 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:01.574972 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:01.575004 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:01.575064 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:08:01.575072 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:01.575165 22182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:01.575205 22182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77476c6004724e3cbbd2e56155707cb6, State: Initialized, Role: FOLLOWER
I20260924 05:08:01.575330 22182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77476c6004724e3cbbd2e56155707cb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260924 05:08:01.575408 22182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:01.575438 22182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:01.575488 22182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:01.575968 22182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77476c6004724e3cbbd2e56155707cb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260924 05:08:01.576061 22182 leader_election.cc:302] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77476c6004724e3cbbd2e56155707cb6; no voters:
I20260924 05:08:01.576205 22182 leader_election.cc:288] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:01.576242 22192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:01.576399 22192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [term 1 LEADER]: Becoming Leader. State: Replica: 77476c6004724e3cbbd2e56155707cb6, State: Running, Role: LEADER
I20260924 05:08:01.576416 22182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:01.576503 22192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77476c6004724e3cbbd2e56155707cb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } }
I20260924 05:08:01.576825 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77476c6004724e3cbbd2e56155707cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77476c6004724e3cbbd2e56155707cb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } } }
I20260924 05:08:01.576833 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77476c6004724e3cbbd2e56155707cb6. Latest consensus state: current_term: 1 leader_uuid: "77476c6004724e3cbbd2e56155707cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77476c6004724e3cbbd2e56155707cb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39927 } } }
I20260924 05:08:01.576984 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:01.576990 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:01.577214 22201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:01.577692 22201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:01.579217 22201 catalog_manager.cc:1347] Generated new cluster ID: 2b51fff9a4a14727a4e05af3f2db1dca
I20260924 05:08:01.579259 22201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:01.591598 22201 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:01.591739 22201 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:01.596596 22201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77476c6004724e3cbbd2e56155707cb6: Generated new TSK 0
I20260924 05:08:01.596745 22201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:08:01.600908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22114
2026-09-24T05:08:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:01 dist-test-slave-m01t krb5kdc[22213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:01 dist-test-slave-m01t krb5kdc[22213](info): set up 2 sockets
Sep 24 05:08:01 dist-test-slave-m01t krb5kdc[22213](info): commencing operation
krb5kdc: starting...
W20260924 05:08:02.537540 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:02 dist-test-slave-m01t krb5kdc[22213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:02.644675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36341
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:02.721483 22226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:02.721695 22226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:02.721740 22226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:02.722873 22226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:02.722913 22226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:02.722924 22226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:02.722935 22226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:02.722945 22226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:02.724349 22226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:02.724547 22226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:02.724720 22226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:02.729250 22231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:02.729270 22232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:02.729302 22226 server_base.cc:1034] running on GCE node
W20260924 05:08:02.729270 22234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:02.729879 22226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:02.730185 22226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:02.731343 22226 hybrid_clock.cc:648] HybridClock initialized: now 1790226482731308 us; error 33 us; skew 500 ppm
Sep 24 05:08:02 dist-test-slave-m01t krb5kdc[22213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:02.739339 22226 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:02.739814 22226 webserver.cc:460] Webserver started at http://127.0.98.62:39345/ using document root <none> and password file <none>
I20260924 05:08:02.739995 22226 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:02.740033 22226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:02.740125 22226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:02.741128 22226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "436fe6dc2b1945ddb4158c7f5bbef774"
format_stamp: "Formatted at 2026-09-24 05:08:02 on dist-test-slave-m01t"
server_key: "de714dac6a4f6cfc4fcf33d7c30e3626"
server_key_iv: "1dff86df810eee76fe13771dd3f52409"
server_key_version: "encryptionkey@0"
I20260924 05:08:02.741427 22226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "436fe6dc2b1945ddb4158c7f5bbef774"
format_stamp: "Formatted at 2026-09-24 05:08:02 on dist-test-slave-m01t"
server_key: "de714dac6a4f6cfc4fcf33d7c30e3626"
server_key_iv: "1dff86df810eee76fe13771dd3f52409"
server_key_version: "encryptionkey@0"
I20260924 05:08:02.742671 22226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:02.743292 22241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:02.743450 22226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:02.743520 22226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "436fe6dc2b1945ddb4158c7f5bbef774"
format_stamp: "Formatted at 2026-09-24 05:08:02 on dist-test-slave-m01t"
server_key: "de714dac6a4f6cfc4fcf33d7c30e3626"
server_key_iv: "1dff86df810eee76fe13771dd3f52409"
server_key_version: "encryptionkey@0"
I20260924 05:08:02.743587 22226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:02.753826 22226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:02.755721 22226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:02.755838 22226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:02.759351 22226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36341
I20260924 05:08:02.759394 22293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36341 every 8 connection(s)
I20260924 05:08:02.759670 22226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:02.760412 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:02.762506 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774: Bootstrap starting.
I20260924 05:08:02.763015 22294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:02.763252 22294 log.cc:824] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:02.763844 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774: No bootstrap required, opened a new log
I20260924 05:08:02.765331 22294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36341 } }
I20260924 05:08:02.765465 22294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:02.765497 22294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436fe6dc2b1945ddb4158c7f5bbef774, State: Initialized, Role: FOLLOWER
I20260924 05:08:02.765623 22294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36341 } }
I20260924 05:08:02.765681 22294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:02.765700 22294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:02.765771 22294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:02.766239 22294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36341 } }
I20260924 05:08:02.766322 22294 leader_election.cc:302] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 436fe6dc2b1945ddb4158c7f5bbef774; no voters:
I20260924 05:08:02.766479 22294 leader_election.cc:288] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:02.766521 22297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:02.766656 22297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [term 1 LEADER]: Becoming Leader. State: Replica: 436fe6dc2b1945ddb4158c7f5bbef774, State: Running, Role: LEADER
I20260924 05:08:02.766695 22294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:02.766741 22297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36341 } }
I20260924 05:08:02.767097 22298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36341 } } }
I20260924 05:08:02.767095 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436fe6dc2b1945ddb4158c7f5bbef774. Latest consensus state: current_term: 1 leader_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436fe6dc2b1945ddb4158c7f5bbef774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36341 } } }
I20260924 05:08:02.767191 22298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:02.767198 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:02.767391 22301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:02.767841 22301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:02.768539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22226
I20260924 05:08:02.768651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:02.768834 392 external_mini_cluster.cc:1428] Setting key f45b6786406546d665e519fde9241c0c
I20260924 05:08:02.769524 22301 catalog_manager.cc:1347] Generated new cluster ID: 1c1f6a5832744744b6eb2adf65dae390
I20260924 05:08:02.769563 22301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:02.773651 22301 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:02.774219 22301 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 05:08:02 dist-test-slave-m01t krb5kdc[22213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:02.778873 22301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 436fe6dc2b1945ddb4158c7f5bbef774: Generated new TSK 0
I20260924 05:08:02.779013 22301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:08:02.779907 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:02.769329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43525 (local address 127.0.98.62:36341)
0924 05:08:02.769490 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:02.769499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:02.769529 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:08:02.769640 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:02.769647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:02.769699 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:02.769860 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:02.770510 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:02.771903 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:02.772894 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:02.773509 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:02.773613 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:02.776341 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:02.776344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:02.776345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:02.776356 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:02.778198 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:02.778564 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:02.778567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:02.778568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:02.778662 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:02.779135 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:02.779136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:02.779137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:02.779340 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:02.779437 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:02.779586 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 05:08:02.779692 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260924 05:08:02.780678 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:02.787299 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:02.782150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36589 (local address 127.0.98.62:36341)
0924 05:08:02.782188 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:02.782190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:02.782198 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:08:02.782251 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:02.782252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:02.782260 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:02.782302 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:02.782750 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:02.783225 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:02.783905 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:02.784124 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:02.784175 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:02.785194 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:02.785195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:02.785196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:02.785202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:02.786358 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:02.786650 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:02.786651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:02.786651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:02.786699 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:02.786852 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:02.786853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:02.786854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:02.786951 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:02.787050 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:02.787153 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 05:08:02.787219 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:08:02.788924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22226
2026-09-24T05:08:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:02 dist-test-slave-m01t krb5kdc[22325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:02 dist-test-slave-m01t krb5kdc[22325](info): set up 2 sockets
Sep 24 05:08:02 dist-test-slave-m01t krb5kdc[22325](info): commencing operation
krb5kdc: starting...
W20260924 05:08:03.729552 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:03 dist-test-slave-m01t krb5kdc[22325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:03.834650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:03.924592 22338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:03.924817 22338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:03.924849 22338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:03.926076 22338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:03.926128 22338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:03.926147 22338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:03.926162 22338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:03.926178 22338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:03.927721 22338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:03.927994 22338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:03.928206 22338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:03.932806 22344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:03.932832 22346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:03.932955 22338 server_base.cc:1034] running on GCE node
W20260924 05:08:03.932816 22343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:03.933254 22338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:03.933555 22338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:03.934697 22338 hybrid_clock.cc:648] HybridClock initialized: now 1790226483934673 us; error 39 us; skew 500 ppm
Sep 24 05:08:03 dist-test-slave-m01t krb5kdc[22325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:03.936983 22338 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:03.937427 22338 webserver.cc:460] Webserver started at http://127.0.98.62:35891/ using document root <none> and password file <none>
I20260924 05:08:03.937634 22338 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:03.937681 22338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:03.937831 22338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:03.938887 22338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "2cdf3ddb075b484197b6a5b57bdfeba8"
format_stamp: "Formatted at 2026-09-24 05:08:03 on dist-test-slave-m01t"
server_key: "6313cb8fa44ed223424244012f646420"
server_key_iv: "6ea45155b5ae0d77cf158ddae60b9105"
server_key_version: "encryptionkey@0"
I20260924 05:08:03.939221 22338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cdf3ddb075b484197b6a5b57bdfeba8"
format_stamp: "Formatted at 2026-09-24 05:08:03 on dist-test-slave-m01t"
server_key: "6313cb8fa44ed223424244012f646420"
server_key_iv: "6ea45155b5ae0d77cf158ddae60b9105"
server_key_version: "encryptionkey@0"
I20260924 05:08:03.940433 22338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:08:03.941071 22353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:03.941218 22338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:03.941298 22338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "2cdf3ddb075b484197b6a5b57bdfeba8"
format_stamp: "Formatted at 2026-09-24 05:08:03 on dist-test-slave-m01t"
server_key: "6313cb8fa44ed223424244012f646420"
server_key_iv: "6ea45155b5ae0d77cf158ddae60b9105"
server_key_version: "encryptionkey@0"
I20260924 05:08:03.941358 22338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:03.951150 22338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:03.952754 22338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:03.952879 22338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:03.956425 22338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44387
I20260924 05:08:03.956458 22405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44387 every 8 connection(s)
I20260924 05:08:03.956758 22338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:03.957535 22406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:03.959125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22338
I20260924 05:08:03.959240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:03.959414 392 external_mini_cluster.cc:1428] Setting key 4939e1a58e64f80968686e2b054e4e0a
I20260924 05:08:03.959906 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8: Bootstrap starting.
I20260924 05:08:03.960440 22406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:03.960686 22406 log.cc:824] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:03.961388 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8: No bootstrap required, opened a new log
I20260924 05:08:03.963495 22406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:08:03.963656 22406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:03.963692 22406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdf3ddb075b484197b6a5b57bdfeba8, State: Initialized, Role: FOLLOWER
I20260924 05:08:03.963838 22406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:08:03.963912 22406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:03.963933 22406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:03.963977 22406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:03.964532 22406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:08:03.964635 22406 leader_election.cc:302] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cdf3ddb075b484197b6a5b57bdfeba8; no voters:
I20260924 05:08:03.964782 22406 leader_election.cc:288] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:03.964821 22411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:03.965021 22411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [term 1 LEADER]: Becoming Leader. State: Replica: 2cdf3ddb075b484197b6a5b57bdfeba8, State: Running, Role: LEADER
I20260924 05:08:03.965029 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:03.959890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55739 (local address 127.0.98.62:44387)
0924 05:08:03.960088 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:03.960097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:03.960120 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:08:03.960538 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:03.960545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:03.960602 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:03.960723 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:03.960948 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:03.962274 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:03.963061 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:03.963647 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:03.963756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:03.964390 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:03.964394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:03.964394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:03.964590 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:03.964629 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:03.964642 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:03.964707 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:03.964774 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:08:03.964845 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260924 05:08:03.965056 22406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:03.965152 22411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:08:03.965502 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:08:03.965574 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:03.965890 22413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cdf3ddb075b484197b6a5b57bdfeba8. Latest consensus state: current_term: 1 leader_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdf3ddb075b484197b6a5b57bdfeba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:08:03.965998 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:03.966150 22413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:03.966291 22416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:03.966779 22416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:03.968441 22416 catalog_manager.cc:1347] Generated new cluster ID: 8f920e555a56476c823376eca9aabcdb
I20260924 05:08:03.968492 22416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:03.970554 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:03.967416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40921 (local address 127.0.98.62:44387)
0924 05:08:03.967448 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:03.967449 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:03.967569 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:08:03.967642 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:03.967643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:03.967652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:03.967707 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:03.967826 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:03.968372 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:03.969335 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:03.969618 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:03.969661 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:03.970294 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:03.970295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:03.970295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:03.970299 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:03.970312 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:03.970316 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:03.970352 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:03.970470 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:08:03.970487 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:08:03.975939 22416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:03.976436 22416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:03.983821 22416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cdf3ddb075b484197b6a5b57bdfeba8: Generated new TSK 0
I20260924 05:08:03.983973 22416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:08:03.993432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22338
2026-09-24T05:08:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:04 dist-test-slave-m01t krb5kdc[22437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:04 dist-test-slave-m01t krb5kdc[22437](info): set up 2 sockets
Sep 24 05:08:04 dist-test-slave-m01t krb5kdc[22437](info): commencing operation
krb5kdc: starting...
W20260924 05:08:04.933602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:05 dist-test-slave-m01t krb5kdc[22437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:05.034555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:05.110529 22450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:05.110734 22450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:05.110765 22450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:05.111886 22450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:05.111927 22450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:05.111940 22450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:05.111950 22450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:05.111961 22450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:05.113340 22450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:05.113513 22450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:05.113668 22450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:05.118091 22455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:05.118196 22450 server_base.cc:1034] running on GCE node
W20260924 05:08:05.118283 22456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:05.118084 22458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:05.118516 22450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:05.118782 22450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:05.119918 22450 hybrid_clock.cc:648] HybridClock initialized: now 1790226485119907 us; error 27 us; skew 500 ppm
Sep 24 05:08:05 dist-test-slave-m01t krb5kdc[22437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:05.122254 22450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:05.122638 22450 webserver.cc:460] Webserver started at http://127.0.98.62:45565/ using document root <none> and password file <none>
I20260924 05:08:05.122823 22450 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:05.122862 22450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:05.122949 22450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:05.123927 22450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "2c77130b631141f5a1aa1607c9034145"
format_stamp: "Formatted at 2026-09-24 05:08:05 on dist-test-slave-m01t"
server_key: "86cc8fa899ba1b6fcda03e5d74b735e8"
server_key_iv: "c4da3df63f86c63aed77dd3a2a019acc"
server_key_version: "encryptionkey@0"
I20260924 05:08:05.124212 22450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c77130b631141f5a1aa1607c9034145"
format_stamp: "Formatted at 2026-09-24 05:08:05 on dist-test-slave-m01t"
server_key: "86cc8fa899ba1b6fcda03e5d74b735e8"
server_key_iv: "c4da3df63f86c63aed77dd3a2a019acc"
server_key_version: "encryptionkey@0"
I20260924 05:08:05.125365 22450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:08:05.126040 22465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:05.126192 22450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:05.126271 22450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "2c77130b631141f5a1aa1607c9034145"
format_stamp: "Formatted at 2026-09-24 05:08:05 on dist-test-slave-m01t"
server_key: "86cc8fa899ba1b6fcda03e5d74b735e8"
server_key_iv: "c4da3df63f86c63aed77dd3a2a019acc"
server_key_version: "encryptionkey@0"
I20260924 05:08:05.126334 22450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:05.129455 22450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:05.131517 22450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:05.131620 22450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:05.135370 22450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35511
I20260924 05:08:05.135402 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35511 every 8 connection(s)
I20260924 05:08:05.135703 22450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:05.136649 22518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:05.138406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22450
I20260924 05:08:05.138530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:05.138708 392 external_mini_cluster.cc:1428] Setting key ace6a582b3903145e78a14775e9d1fc2
I20260924 05:08:05.139166 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145: Bootstrap starting.
I20260924 05:08:05.139773 22518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:05.140033 22518 log.cc:824] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:05.140725 22518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145: No bootstrap required, opened a new log
I20260924 05:08:05.141445 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:05.139178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44047 (local address 127.0.98.62:35511)
0924 05:08:05.139359 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:05.139367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:05.139443 (+ 76us) server_negotiation.cc:407] Connection header received
0924 05:08:05.139935 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:05.139941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:05.139967 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:05.140081 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:05.141022 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:05.141025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:05.141026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:05.141029 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:05.141064 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:05.141073 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:05.141139 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:05.141201 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 05:08:05.141233 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 05:08:05.142721 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:05.143374 22518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c77130b631141f5a1aa1607c9034145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:08:05.143553 22518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:05.143594 22518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c77130b631141f5a1aa1607c9034145, State: Initialized, Role: FOLLOWER
I20260924 05:08:05.143698 22518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c77130b631141f5a1aa1607c9034145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:08:05.143781 22518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:05.143810 22518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:05.143851 22518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:05.144457 22518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c77130b631141f5a1aa1607c9034145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:08:05.144572 22518 leader_election.cc:302] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c77130b631141f5a1aa1607c9034145; no voters:
I20260924 05:08:05.144718 22518 leader_election.cc:288] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:05.144871 22529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:05.144955 22518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:05.145049 22529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [term 1 LEADER]: Becoming Leader. State: Replica: 2c77130b631141f5a1aa1607c9034145, State: Running, Role: LEADER
I20260924 05:08:05.145164 22529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c77130b631141f5a1aa1607c9034145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } }
I20260924 05:08:05.145536 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c77130b631141f5a1aa1607c9034145. Latest consensus state: current_term: 1 leader_uuid: "2c77130b631141f5a1aa1607c9034145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c77130b631141f5a1aa1607c9034145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } } }
I20260924 05:08:05.145623 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:05.145790 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c77130b631141f5a1aa1607c9034145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c77130b631141f5a1aa1607c9034145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35511 } } }
I20260924 05:08:05.145838 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:05.144660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53085 (local address 127.0.98.62:35511)
0924 05:08:05.144690 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:05.144692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:05.144812 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:08:05.144879 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:05.144879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:05.144883 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:05.144937 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:05.145595 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:05.145596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:05.145596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:08:05.145600 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:05.145611 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:08:05.145615 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:05.145643 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:05.145743 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 05:08:05.145756 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 05:08:05.145856 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:05.146351 22537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:05.146807 22537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:05.148027 22537 catalog_manager.cc:1347] Generated new cluster ID: 0b73add7cc2242059c87fb18797c4763
I20260924 05:08:05.148077 22537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:05.154958 22537 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:05.155085 22537 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:05.159003 22537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c77130b631141f5a1aa1607c9034145: Generated new TSK 0
I20260924 05:08:05.159138 22537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:08:05.162248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22450
2026-09-24T05:08:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1168 ms)
[----------] 18 tests from AuthTokenIssuingTest (21494 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:05 dist-test-slave-m01t krb5kdc[22549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:05 dist-test-slave-m01t krb5kdc[22549](info): set up 2 sockets
Sep 24 05:08:05 dist-test-slave-m01t krb5kdc[22549](info): commencing operation
krb5kdc: starting...
W20260924 05:08:06.097507 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:06.202409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40039
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40039,127.0.98.61:36519,127.0.98.60:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:06.279467 22562 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:08:06.279654 22562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:06.279670 22562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:06.279688 22562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:06.280718 22562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:06.280750 22562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:06.280762 22562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:06.280773 22562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:06.280788 22562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:06.282179 22562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40039,127.0.98.61:36519,127.0.98.60:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:06.282356 22562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:06.282512 22562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:06.286804 22567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:06.286804 22570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:06.286891 22568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:06.286917 22562 server_base.cc:1034] running on GCE node
I20260924 05:08:06.287230 22562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:06.287518 22562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:06.288657 22562 hybrid_clock.cc:648] HybridClock initialized: now 1790226486288643 us; error 28 us; skew 500 ppm
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:06.291239 22562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:06.291658 22562 webserver.cc:460] Webserver started at http://127.0.98.62:41559/ using document root <none> and password file <none>
I20260924 05:08:06.291850 22562 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:06.291894 22562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:06.292014 22562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:06.293010 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "48a0f3f16f454b1dbb30d972244f4ab4"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "357eca021013558e47de08b57cf5b4d0"
server_key_iv: "c4f12a4765c99decc6cede3b97975e69"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.293326 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "48a0f3f16f454b1dbb30d972244f4ab4"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "357eca021013558e47de08b57cf5b4d0"
server_key_iv: "c4f12a4765c99decc6cede3b97975e69"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.294569 22562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:06.295176 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:06.295315 22562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:06.295394 22562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "48a0f3f16f454b1dbb30d972244f4ab4"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "357eca021013558e47de08b57cf5b4d0"
server_key_iv: "c4f12a4765c99decc6cede3b97975e69"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.295450 22562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:06.302551 22562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:06.304069 22562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:06.304188 22562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:06.307727 22562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40039
I20260924 05:08:06.307754 22629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40039 every 8 connection(s)
I20260924 05:08:06.308051 22562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:06.308894 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:06.310322 22630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:06.316073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22562
I20260924 05:08:06.316175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:06.316339 392 external_mini_cluster.cc:1428] Setting key 1f54e0283a397fa46df4229f56df9efa
I20260924 05:08:06.318629 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.310904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.0.98.62:40039)
0924 05:08:06.311047 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.311053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.311118 (+ 65us) server_negotiation.cc:407] Connection header received
0924 05:08:06.311568 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.311570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.311585 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.311651 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.312086 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.312900 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.313692 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.314030 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.314103 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.315760 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.315767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.315767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.315773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.317200 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.317547 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.317550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.317550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.317622 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.317841 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.317842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.317842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.318041 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.318109 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.318269 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.318375 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260924 05:08:06.318630 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.310867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40039 (local address 127.0.0.1:56470)
0924 05:08:06.311010 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.311044 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.311466 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.311656 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.311659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.311868 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.312026 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.312029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.312980 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.312981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.313632 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.313634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.314106 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.314107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.314159 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.314517 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.314524 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.315692 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.317292 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.317297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.317299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.317480 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.317675 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.317676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.317677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.317781 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.318099 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.318103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.318162 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.318242 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.318443 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":74,"thread_start_us":64,"threads_started":1}
I20260924 05:08:06.319720 22630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:06.320041 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.319911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39718)
0924 05:08:06.319959 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.320011 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:08:06.320176 22579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
I20260924 05:08:06.320572 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.320479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39730)
0924 05:08:06.320521 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.320550 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 05:08:06.320672 22630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36519: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:08:06.337131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36519
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40039,127.0.98.61:36519,127.0.98.60:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:08:06.369874 22630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } attempt: 1
I20260924 05:08:06.370385 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.370200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39738)
0924 05:08:06.370260 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.370312 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 05:08:06.370733 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.370630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39748)
0924 05:08:06.370677 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.370708 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:08:06.370836 22630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36519: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
W20260924 05:08:06.414220 22637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:08:06.414415 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:06.414431 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:06.414449 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:06.415549 22637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:06.415591 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:06.415603 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:06.415614 22637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:06.415624 22637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:06.417054 22637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40039,127.0.98.61:36519,127.0.98.60:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:06.417269 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:06.417493 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:06.422073 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:06.422106 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:06.422084 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:06.422439 22637 server_base.cc:1034] running on GCE node
I20260924 05:08:06.422653 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:06.422945 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:06.424081 22637 hybrid_clock.cc:648] HybridClock initialized: now 1790226486424067 us; error 30 us; skew 500 ppm
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:06.426402 22637 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:08:06.426821 22637 webserver.cc:460] Webserver started at http://127.0.98.61:40937/ using document root <none> and password file <none>
I20260924 05:08:06.427014 22637 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:06.427057 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:06.427175 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:06.428211 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data/instance:
uuid: "62531054a62c415c903b3a6e1be231ab"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "4323cfcfb654b85d78e021f2167ee66a"
server_key_iv: "21ff15cda5b8e2b8a343c60b3194b93a"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.428544 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal/instance:
uuid: "62531054a62c415c903b3a6e1be231ab"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "4323cfcfb654b85d78e021f2167ee66a"
server_key_iv: "21ff15cda5b8e2b8a343c60b3194b93a"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.429813 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:08:06.430474 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:06.430644 22637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:06.430723 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal
uuid: "62531054a62c415c903b3a6e1be231ab"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "4323cfcfb654b85d78e021f2167ee66a"
server_key_iv: "21ff15cda5b8e2b8a343c60b3194b93a"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.430794 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:06.438979 22630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } attempt: 2
I20260924 05:08:06.439402 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.439268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39760)
0924 05:08:06.439320 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.439373 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:08:06.439735 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.439652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39772)
0924 05:08:06.439691 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.439715 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:08:06.439808 22630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36519: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36519: connect: Connection refused (error 111)
I20260924 05:08:06.450405 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:06.452071 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:06.452199 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:06.455880 22637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36519
I20260924 05:08:06.455917 22704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36519 every 8 connection(s)
I20260924 05:08:06.456233 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:08:06.457113 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:06.458673 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:06.461297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22637
I20260924 05:08:06.461380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-1/wal/instance
I20260924 05:08:06.461526 392 external_mini_cluster.cc:1428] Setting key 6909e5e59c7e927752ca0bd83c54cc40
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:06.467087 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.459293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.98.62:40039)
0924 05:08:06.459355 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.459358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.459531 (+ 173us) server_negotiation.cc:407] Connection header received
0924 05:08:06.459953 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.459953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.459973 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.460022 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.460562 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.461027 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.462080 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.462278 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.462335 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.464447 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.464449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.464449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.464457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.465907 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.466252 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.466254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.466254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.466316 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.466505 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.466506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.466506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.466641 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.466704 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.466909 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.466995 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:08:06.467126 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.459267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40039 (local address 127.0.0.1:56480)
0924 05:08:06.459422 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.459455 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.459880 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.460080 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.460083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.460308 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.460499 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.460503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.461095 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.461096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.461997 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.461999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.462369 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.462370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.462472 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.462945 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.462956 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.464355 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.465976 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.465982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.465984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.466186 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.466371 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.466372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.466372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.466454 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.466693 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.466696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.466798 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.466872 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.466978 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260924 05:08:06.467676 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:06.474448 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.467904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39780)
0924 05:08:06.467941 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.467951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.467987 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.468207 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.468208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.468324 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.468365 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.468367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.469166 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.469167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.469746 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.469748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.470221 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.470222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.470279 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.470612 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.470617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.471889 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.473418 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.473419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.473420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.473629 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.473831 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.473832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.473832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.473896 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.474126 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.474127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.474174 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.474249 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.474344 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:08:06.474519 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.467973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.0.98.61:36519)
0924 05:08:06.468096 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.468101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.468116 (+ 15us) server_negotiation.cc:407] Connection header received
0924 05:08:06.468131 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.468132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.468141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.468197 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.468426 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.469099 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.469813 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.470159 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.470215 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.471970 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.471979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.471979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.471987 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.473332 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.473696 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.473699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.473699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.473770 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.473950 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.473950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.473950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.474070 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.474127 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.474271 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.474401 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260924 05:08:06.475270 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:06.475589 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.475473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35708)
0924 05:08:06.475516 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.475564 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:08:06.475725 22656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
I20260924 05:08:06.476089 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.475994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35724)
0924 05:08:06.476037 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.476068 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:08:06.476188 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:08:06.483443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45703
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40039,127.0.98.61:36519,127.0.98.60:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:08:06.525424 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } attempt: 1
I20260924 05:08:06.525965 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.525805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35730)
0924 05:08:06.525872 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.525922 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 05:08:06.526290 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.526212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35734)
0924 05:08:06.526248 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.526272 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:08:06.526388 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
I20260924 05:08:06.530937 22630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } attempt: 3
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:06.537679 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.531238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39796)
0924 05:08:06.531290 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.531302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.531350 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.531475 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.531476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.531638 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.531705 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.531706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.532363 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.532365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.532930 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.532934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.533277 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.533278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.533358 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.533867 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.533872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.535254 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.536828 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.536829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.536829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.536984 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.537159 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.537160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.537160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.537204 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.537453 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.537454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.537492 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.537537 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.537608 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:08:06.537756 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.531309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.0.98.61:36519)
0924 05:08:06.531370 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.531372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.531383 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:08:06.531409 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.531410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.531421 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.531465 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.531779 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.532270 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.533002 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.533216 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.533272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.535340 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.535342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.535342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.535351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.536751 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.537044 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.537045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.537046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.537105 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.537248 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.537249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.537249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.537394 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.537470 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.537563 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.537641 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:08:06.538201 22630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:06.538538 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.538416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35750)
0924 05:08:06.538471 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.538508 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:08:06.538869 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.538782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35758)
0924 05:08:06.538827 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.538851 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:08:06.538972 22630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
W20260924 05:08:06.562734 22712 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:08:06.562912 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:06.562927 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:06.562945 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:06.564013 22712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:06.564043 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:06.564055 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:06.564066 22712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:06.564079 22712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:06.565450 22712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40039,127.0.98.61:36519,127.0.98.60:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:06.565622 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:06.565821 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:06.569995 22718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:06.570089 22630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } attempt: 1
I20260924 05:08:06.570137 22712 server_base.cc:1034] running on GCE node
W20260924 05:08:06.569978 22717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:06.570011 22720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:06.570407 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.570299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35774)
0924 05:08:06.570342 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.570376 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:08:06.570576 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:06.570729 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.570626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35778)
0924 05:08:06.570674 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.570703 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:08:06.570873 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260924 05:08:06.570873 22630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
I20260924 05:08:06.572017 22712 hybrid_clock.cc:648] HybridClock initialized: now 1790226486572003 us; error 28 us; skew 500 ppm
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:06.574400 22712 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:08:06.574877 22712 webserver.cc:460] Webserver started at http://127.0.98.60:44263/ using document root <none> and password file <none>
I20260924 05:08:06.575080 22712 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:06.575127 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:06.575244 22712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:06.576196 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data/instance:
uuid: "f293787e8b5f48a49e4129245566154b"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "91c771866e26d724dd42c2429618e92c"
server_key_iv: "2802b4a93ac7482e8524aba05fbae50b"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.576507 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal/instance:
uuid: "f293787e8b5f48a49e4129245566154b"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "91c771866e26d724dd42c2429618e92c"
server_key_iv: "2802b4a93ac7482e8524aba05fbae50b"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.577684 22712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:08:06.578379 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:06.578550 22712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:06.578629 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal
uuid: "f293787e8b5f48a49e4129245566154b"
format_stamp: "Formatted at 2026-09-24 05:08:06 on dist-test-slave-m01t"
server_key: "91c771866e26d724dd42c2429618e92c"
server_key_iv: "2802b4a93ac7482e8524aba05fbae50b"
server_key_version: "encryptionkey@0"
I20260924 05:08:06.578692 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:06.594537 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } attempt: 2
I20260924 05:08:06.594784 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:06.594898 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.594762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35786)
0924 05:08:06.594826 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.594865 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 05:08:06.595224 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.595132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35802)
0924 05:08:06.595171 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.595198 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:08:06.595330 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45703: connect: Connection refused (error 111)
I20260924 05:08:06.596860 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:06.596977 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:06.600621 22712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45703
I20260924 05:08:06.600656 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45703 every 8 connection(s)
I20260924 05:08:06.600966 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:08:06.601857 22780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:06.603269 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:06.607460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22712
I20260924 05:08:06.607551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226228698807-392-0/minicluster-data/master-2/wal/instance
I20260924 05:08:06.607741 392 external_mini_cluster.cc:1428] Setting key bbed5bac440cfd0ef768e868bc32c306
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:06.611057 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.603857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.0.98.62:40039)
0924 05:08:06.603911 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.603913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.604089 (+ 176us) server_negotiation.cc:407] Connection header received
0924 05:08:06.604534 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.604535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.604543 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.604581 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.605122 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.605566 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.606466 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.606637 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.606679 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.608614 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.608616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.608616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.608623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.609893 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.610237 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.610239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.610239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.610292 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.610493 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.610494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.610494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.610602 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.610657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.610900 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.610972 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:08:06.611212 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.603836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40039 (local address 127.0.0.1:56482)
0924 05:08:06.603981 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.604019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.604447 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.604640 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.604643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.604884 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.605063 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.605067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.605618 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.605619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.606401 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.606403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.606700 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.606701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.606803 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.607161 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.607168 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.608536 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.609969 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.609975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.609977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.610175 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.610343 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.610344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.610345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.610440 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.610655 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.610660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.610789 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.610872 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.611010 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260924 05:08:06.611716 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:06.615940 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.608089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.0.98.60:45703)
0924 05:08:06.608234 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.608238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.608609 (+ 371us) server_negotiation.cc:407] Connection header received
0924 05:08:06.608624 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.608625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.608635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.608683 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.609374 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.610090 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.610760 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.611035 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.611091 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.613361 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.613368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.613368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.613374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.614898 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.615221 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.615224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.615224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.615278 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.615434 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.615435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.615435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.615582 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.615711 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.615808 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.615871 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260924 05:08:06.617705 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.611991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.98.61:36519)
0924 05:08:06.612040 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.612042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.612070 (+ 28us) server_negotiation.cc:407] Connection header received
0924 05:08:06.612118 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.612119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.612129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.612190 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.612418 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.612903 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.613591 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.613849 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.613915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.615366 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.615367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.615368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.615374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.616787 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.617061 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.617062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.617062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.617115 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.617266 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.617267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.617267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.617382 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.617436 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.617559 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.617623 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:08:06.617758 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.611938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36519 (local address 127.0.0.1:39798)
0924 05:08:06.612029 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.612037 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.612077 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.612191 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.612192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.612318 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.612358 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.612359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.612973 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.612974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.613529 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.613531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.613913 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.613914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.613966 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.614265 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.614269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.615289 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.616853 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.616854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.616854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.617004 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.617164 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.617165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.617165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.617218 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.617434 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.617435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.617472 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.617531 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.617610 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 05:08:06.618194 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:06.623530 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.616686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.0.98.62:40039)
0924 05:08:06.616723 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.616724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.616762 (+ 38us) server_negotiation.cc:407] Connection header received
0924 05:08:06.616815 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.616816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.616825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.616866 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.617284 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.617746 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.618558 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.618819 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.618872 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.620993 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.620994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.620994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.621001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.622415 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.622763 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.622764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.622764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.622819 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.622979 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.622980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.622981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.623102 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.623279 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.623315 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.623468 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:08:06.624181 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.618397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35814)
0924 05:08:06.618482 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.618491 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.618552 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.618649 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.618650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.618764 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.618786 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.618787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.619587 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.619588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.620139 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.620142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.620460 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.620461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.620516 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.620913 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.620917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.621930 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.623398 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.623399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.623399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.623540 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.623695 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.623695 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.623696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.623743 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.623930 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.623931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.623965 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.624025 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.624113 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 05:08:06.624308 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.618440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.0.98.60:45703)
0924 05:08:06.618472 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.618473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.618522 (+ 49us) server_negotiation.cc:407] Connection header received
0924 05:08:06.618597 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.618598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.618607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.618648 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.618841 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.619490 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.620214 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.620396 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.620444 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.622008 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.622009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.622009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.622015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.623321 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.623600 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.623601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.623602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.623646 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.623791 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.623791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.623791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.623882 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.623933 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.624058 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.624223 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":17}
I20260924 05:08:06.625794 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: Bootstrap starting.
I20260924 05:08:06.626605 22780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:06.626859 22780 log.cc:824] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:06.627583 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: No bootstrap required, opened a new log
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:06.629145 22780 raft_consensus.cc:348] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.629272 22780 raft_consensus.cc:729] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f293787e8b5f48a49e4129245566154b, State: Initialized, Role: FOLLOWER
I20260924 05:08:06.629335 22780 consensus_queue.cc:260] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.629650 22780 sys_catalog.cc:564] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:06.629635 22786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.629843 22786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.630980 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.624193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.0.98.61:36519)
0924 05:08:06.624235 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.624237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.624245 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:08:06.624269 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.624270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.624280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.624349 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.624822 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.625292 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.625980 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.626187 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.626239 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.628356 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.628358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.628358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.628365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.629761 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.630151 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.630152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.630153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.630222 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.630504 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.630505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.630506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.630616 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.630763 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.630808 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.630888 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:08:06.631871 22797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:08:06.631979 22797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:08:06.632462 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:06.631888 22712 master_runner.cc:186] Error getting master registration for 127.0.98.62:40039: OK, instance_id { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" instance_seqno: 1790226486306761 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:08:06.634508 22712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:08:06.633690 22604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "48a0f3f16f454b1dbb30d972244f4ab4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492
I20260924 05:08:06.645995 22630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } attempt: 2
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:06.652495 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.646349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.0.98.60:45703)
0924 05:08:06.646389 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.646391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.646430 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:08:06.646505 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.646506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.646514 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.646556 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.646824 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.647281 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.647895 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.648078 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.648131 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.649884 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.649885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.649885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.649892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.651377 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.651741 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.651742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.651742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.651792 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.651966 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.651966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.651966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.652122 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.652182 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.652323 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.652393 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:08:06.652495 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.646321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35822)
0924 05:08:06.646377 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.646389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.646448 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.646577 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.646577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.646723 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.646763 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.646765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.647347 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.647348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.647826 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.647828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.648132 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.648133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.648184 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.648578 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.648582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.649805 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.651454 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.651455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.651455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.651675 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.651849 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.651850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.651850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.651913 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.652179 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.652179 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.652229 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.652301 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.652396 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:08:06.654030 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4: Bootstrap starting.
I20260924 05:08:06.654630 22630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:06.654894 22630 log.cc:824] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:06.655560 22630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4: No bootstrap required, opened a new log
I20260924 05:08:06.656514 22630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.656625 22630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a0f3f16f454b1dbb30d972244f4ab4, State: Initialized, Role: FOLLOWER
I20260924 05:08:06.656701 22630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.656993 22630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:06.657048 22604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 05:08:06.657126 22604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:06.656949 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.657838 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.657884 22604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.658344 22754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f293787e8b5f48a49e4129245566154b"
I20260924 05:08:06.658471 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [term 0 FOLLOWER]: Advancing to term 1
W20260924 05:08:06.658977 22811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:08:06.659031 22811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:08:06.659287 22754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a0f3f16f454b1dbb30d972244f4ab4 in term 1.
I20260924 05:08:06.659969 22581 leader_election.cc:302] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a0f3f16f454b1dbb30d972244f4ab4, f293787e8b5f48a49e4129245566154b; no voters:
I20260924 05:08:06.660174 22800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:06.660310 22800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 48a0f3f16f454b1dbb30d972244f4ab4, State: Running, Role: LEADER
I20260924 05:08:06.660391 22800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.661181 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48a0f3f16f454b1dbb30d972244f4ab4. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.661312 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:06.661547 22815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:06.661789 22815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:06.661268 22562 master_runner.cc:186] Error getting master registration for 127.0.98.61:36519: OK, instance_id { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" instance_seqno: 1790226486454861 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:08:06.663506 22562 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:08:06.663587 22604 leader_election.cc:288] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 62531054a62c415c903b3a6e1be231ab (127.0.98.61:36519), f293787e8b5f48a49e4129245566154b (127.0.98.60:45703)
I20260924 05:08:06.664793 22754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [term 1 FOLLOWER]: Refusing update from remote peer 48a0f3f16f454b1dbb30d972244f4ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:06.665103 22813 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:06.666370 22786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48a0f3f16f454b1dbb30d972244f4ab4. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.666460 22786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.667552 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.667629 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:06.668038 22786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.668116 22786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.668431 22815 catalog_manager.cc:1347] Generated new cluster ID: 00e50bf00eba4c8882ac06f4cabce9fe
I20260924 05:08:06.668489 22815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:06.686514 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } attempt: 3
I20260924 05:08:06.688370 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.670173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.98.61:36519)
0924 05:08:06.670214 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.670216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.677982 (+ 7766us) server_negotiation.cc:407] Connection header received
0924 05:08:06.678056 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.678057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.678067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.678110 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.679188 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.679859 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.681508 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.682294 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.682333 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.684857 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.684858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.684859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.684866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.686603 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.687196 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.687197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.687210 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.687269 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.687602 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.687603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.687603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.687731 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.687813 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.688199 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.688276 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:06.690088 22815 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:06.690586 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.669958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.98.62:40039)
0924 05:08:06.669996 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.669998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.677866 (+ 7868us) server_negotiation.cc:407] Connection header received
0924 05:08:06.678083 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.678084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.678093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.678121 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.678806 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.679492 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.682154 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.682414 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.682450 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.685758 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.685759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.685760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.685767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.687764 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.689789 (+ 2025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.689791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.689791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.689852 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.690050 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.690051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.690051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.690169 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.690243 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.690424 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.690501 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:08:06.690994 22815 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:06.693904 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.670555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.98.60:45703)
0924 05:08:06.673774 (+ 3219us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.673776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.680506 (+ 6730us) server_negotiation.cc:407] Connection header received
0924 05:08:06.680591 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.680592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.680601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.680626 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.681347 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.682029 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.684664 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.684940 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.684979 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.691206 (+ 6227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.691207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.691207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.691213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.692730 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.693104 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.693105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.693105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.693157 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.693345 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.693346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.693346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.693464 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.693527 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.693738 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.693819 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3200}
I20260924 05:08:06.696249 22815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4: Generated new TSK 0
I20260924 05:08:06.696451 22815 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 24 05:08:06 dist-test-slave-m01t krb5kdc[22549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:06.701102 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.686731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45703 (local address 127.0.0.1:35848)
0924 05:08:06.686819 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:06.686829 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:06.686864 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:06.694123 (+ 7259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:06.694124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:06.694268 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:06.694307 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.694309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.695140 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.695141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.695911 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:06.695913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.696365 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.696366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:06.696419 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.696875 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:06.696880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:06.698397 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:06.700134 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.700135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.700136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.700326 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.700518 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:06.700519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:06.700519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:06.700573 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:06.700822 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:06.700822 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:06.700863 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:08:06.700919 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:08:06.701004 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 05:08:06.701546 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:06.686945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.0.98.60:45703)
0924 05:08:06.693974 (+ 7029us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.693975 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.693982 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:08:06.693999 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.693999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.694007 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.694054 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.694380 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.695057 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.695995 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:06.696271 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:06.696321 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:06.698527 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:06.698528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:06.698528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:06.698534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:06.700050 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.700395 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.700396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.700397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.700444 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:06.700633 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:06.700634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:06.700634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:06.700747 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:06.700798 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:06.701380 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 05:08:06.701446 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7016,"thread_start_us":86,"threads_started":1}
I20260924 05:08:06.703675 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: Bootstrap starting.
I20260924 05:08:06.704398 22705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:06.704685 22705 log.cc:824] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:06.707291 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: No bootstrap required, opened a new log
I20260924 05:08:06.709187 22705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.709312 22705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62531054a62c415c903b3a6e1be231ab, State: Initialized, Role: FOLLOWER
I20260924 05:08:06.709383 22705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } }
I20260924 05:08:06.709681 22705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:06.709846 22679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62531054a62c415c903b3a6e1be231ab"
I20260924 05:08:06.709973 22678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:06.710932 22678 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [term 1 FOLLOWER]: Refusing update from remote peer 48a0f3f16f454b1dbb30d972244f4ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:06.711202 22800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:06.712129 22679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a0f3f16f454b1dbb30d972244f4ab4 in term 1.
I20260924 05:08:06.713797 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.713886 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.713939 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48a0f3f16f454b1dbb30d972244f4ab4. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.713994 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.714025 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:06.714066 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:06.719033 22834 mvcc.cc:205] Tried to move back new op lower bound from 7332767689503248384 to 7332767689361608704. Current Snapshot: MvccSnapshot[applied={T|T < 7332767689376604160}]
W20260924 05:08:06.719805 22847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:08:06.719936 22847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 05:08:06.720989 22784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.719669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.0.98.60:45703)
0924 05:08:06.719709 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.719711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.720384 (+ 673us) server_negotiation.cc:407] Connection header received
0924 05:08:06.720433 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.720434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.720445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.720509 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.720928 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35860: BlockingRecv error: recv got EOF from 127.0.0.1:35860 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:08:06.721380 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.720237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.98.61:36519)
0924 05:08:06.720271 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.720273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.720816 (+ 543us) server_negotiation.cc:407] Connection header received
0924 05:08:06.720903 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.720904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.720913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.720939 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.721326 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39820: BlockingRecv error: recv got EOF from 127.0.0.1:39820 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:08:06.721918 22633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.720566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.98.62:40039)
0924 05:08:06.720601 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.720602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.721272 (+ 670us) server_negotiation.cc:407] Connection header received
0924 05:08:06.721358 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.721359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.721370 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.721396 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.721861 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56506: BlockingRecv error: recv got EOF from 127.0.0.1:56506 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:08:06.726624 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.725997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.98.61:36519)
0924 05:08:06.726035 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.726037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.726139 (+ 102us) server_negotiation.cc:407] Connection header received
0924 05:08:06.726233 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.726234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.726250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.726276 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.726581 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39824: BlockingRecv error: recv got EOF from 127.0.0.1:39824 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:08:06.729658 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.728818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.0.98.61:36519)
0924 05:08:06.728851 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.728852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.729086 (+ 234us) server_negotiation.cc:407] Connection header received
0924 05:08:06.729175 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.729176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.729186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.729212 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.729611 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39832: BlockingRecv error: recv got EOF from 127.0.0.1:39832 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:08:06.733685 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.732956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.98.61:36519)
0924 05:08:06.732991 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.732993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.733215 (+ 222us) server_negotiation.cc:407] Connection header received
0924 05:08:06.733303 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.733305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.733314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.733342 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.733639 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39834: BlockingRecv error: recv got EOF from 127.0.0.1:39834 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:08:06.739761 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.738979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.0.98.61:36519)
0924 05:08:06.739014 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.739016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.739249 (+ 233us) server_negotiation.cc:407] Connection header received
0924 05:08:06.739344 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.739345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.739355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.739382 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.739713 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39844: BlockingRecv error: recv got EOF from 127.0.0.1:39844 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:08:06.749686 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.748913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.0.98.61:36519)
0924 05:08:06.748947 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.748948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.749179 (+ 231us) server_negotiation.cc:407] Connection header received
0924 05:08:06.749272 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.749273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.749283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.749312 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.749638 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39854: BlockingRecv error: recv got EOF from 127.0.0.1:39854 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:08:06.767578 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.766810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.0.98.61:36519)
0924 05:08:06.766842 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.766845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.767088 (+ 243us) server_negotiation.cc:407] Connection header received
0924 05:08:06.767181 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.767182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.767193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.767220 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.767532 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39860: BlockingRecv error: recv got EOF from 127.0.0.1:39860 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:08:06.801426 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.800782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.98.61:36519)
0924 05:08:06.800827 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.800830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.800920 (+ 90us) server_negotiation.cc:407] Connection header received
0924 05:08:06.800973 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.800974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.800986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.801033 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.801364 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39868: BlockingRecv error: recv got EOF from 127.0.0.1:39868 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:08:06.867177 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.866523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.0.98.61:36519)
0924 05:08:06.866570 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.866572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.866652 (+ 80us) server_negotiation.cc:407] Connection header received
0924 05:08:06.866715 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.866716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.866725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.866776 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.867117 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39882: BlockingRecv error: recv got EOF from 127.0.0.1:39882 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 05:08:06.996784 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:06.996213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.98.61:36519)
0924 05:08:06.996267 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:06.996269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:06.996334 (+ 65us) server_negotiation.cc:407] Connection header received
0924 05:08:06.996392 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:06.996393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:06.996403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:06.996462 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:06.996732 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39894: BlockingRecv error: recv got EOF from 127.0.0.1:39894 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:08:07.233047 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a0f3f16f454b1dbb30d972244f4ab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40039 } } peers { permanent_uuid: "62531054a62c415c903b3a6e1be231ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36519 } } peers { permanent_uuid: "f293787e8b5f48a49e4129245566154b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45703 } } }
I20260924 05:08:07.233186 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a0f3f16f454b1dbb30d972244f4ab4 [sys.catalog]: This master's current role is: LEADER
W20260924 05:08:07.254448 22709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:07.253886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.0.98.61:36519)
0924 05:08:07.253946 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:07.253947 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:07.253996 (+ 49us) server_negotiation.cc:407] Connection header received
0924 05:08:07.254066 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:07.254067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:07.254077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:07.254121 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:07.254397 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39896: BlockingRecv error: recv got EOF from 127.0.0.1:39896 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:08:07.632486 22797 catalog_manager.cc:1259] Loaded cluster ID: 00e50bf00eba4c8882ac06f4cabce9fe
I20260924 05:08:07.632570 22797 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: loading cluster ID for follower catalog manager: success
I20260924 05:08:07.633448 22797 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: acquiring CA information for follower catalog manager: success
I20260924 05:08:07.633783 22797 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f293787e8b5f48a49e4129245566154b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:08:07.720399 22847 catalog_manager.cc:1259] Loaded cluster ID: 00e50bf00eba4c8882ac06f4cabce9fe
I20260924 05:08:07.720475 22847 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: loading cluster ID for follower catalog manager: success
I20260924 05:08:07.721268 22847 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: acquiring CA information for follower catalog manager: success
I20260924 05:08:07.721558 22847 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62531054a62c415c903b3a6e1be231ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:08:07.769932 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:07.767500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.0.98.61:36519)
0924 05:08:07.767622 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:07.767624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:07.767634 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:07.767665 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:07.767665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:07.767728 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:07.767794 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:07.767934 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.768434 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.769181 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.769374 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.769429 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:07.769591 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:07.769701 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:07.769786 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:07.769822 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 05:08:07.769843 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260924 05:08:07.773789 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:07.771558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.0.98.60:45703)
0924 05:08:07.771661 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:07.771663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:07.771681 (+ 18us) server_negotiation.cc:407] Connection header received
0924 05:08:07.771720 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:07.771721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:07.771760 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:07.771824 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:07.771923 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.772402 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.773078 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.773281 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.773331 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:07.773464 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:07.773557 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:07.773618 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:07.773665 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:08:07.773684 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260924 05:08:07.777752 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:07.775102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.98.60:45703)
0924 05:08:07.775200 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:07.775202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:07.775326 (+ 124us) server_negotiation.cc:407] Connection header received
0924 05:08:07.775411 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:07.775412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:07.775419 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:07.775474 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:07.775581 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.776049 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.777029 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.777311 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.777370 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:07.777533 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:07.777563 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:07.777608 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:07.777651 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 05:08:07.777669 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 05:08:07.777980 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:07.775066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.0.98.61:36519)
0924 05:08:07.775169 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:07.775171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:07.775313 (+ 142us) server_negotiation.cc:407] Connection header received
0924 05:08:07.775357 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:07.775358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:07.775367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:07.775408 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:07.775556 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.776267 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.777193 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.777435 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.777486 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:07.777651 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:07.777680 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:07.777705 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:07.777886 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:08:07.777907 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 05:08:07.778379 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:07.775084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.0.98.62:40039)
0924 05:08:07.775247 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:07.775249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:07.775258 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:08:07.775306 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:07.775307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:07.775356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:07.775424 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:07.775611 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.776358 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.777080 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:07.777386 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:07.777440 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:07.777621 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:07.777749 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:07.777821 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:07.778271 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 05:08:07.778301 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":47,"threads_started":1}
I20260924 05:08:07.783412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22562
I20260924 05:08:07.788229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22637
I20260924 05:08:07.792234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22712
2026-09-24T05:08:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2631 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:07 dist-test-slave-m01t krb5kdc[22934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:07 dist-test-slave-m01t krb5kdc[22934](info): set up 2 sockets
Sep 24 05:08:07 dist-test-slave-m01t krb5kdc[22934](info): commencing operation
krb5kdc: starting...
W20260924 05:08:08.749653 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:08 dist-test-slave-m01t krb5kdc[22934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:08.856683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45747,127.0.98.61:46119,127.0.98.60:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:08.933693 22947 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:08:08.933940 22947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:08.933965 22947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:08.933991 22947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:08.935135 22947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:08.935176 22947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:08.935195 22947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:08.935212 22947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:08.935226 22947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:08.936712 22947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45747,127.0.98.61:46119,127.0.98.60:35797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:08.936945 22947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:08.937160 22947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:08.941583 22952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:08.941658 22953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:08.941583 22955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:08.941845 22947 server_base.cc:1034] running on GCE node
I20260924 05:08:08.942080 22947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:08.942346 22947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:08.943483 22947 hybrid_clock.cc:648] HybridClock initialized: now 1790226488943470 us; error 26 us; skew 500 ppm
Sep 24 05:08:08 dist-test-slave-m01t krb5kdc[22934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:08.945752 22947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:08.946135 22947 webserver.cc:460] Webserver started at http://127.0.98.62:44115/ using document root <none> and password file <none>
I20260924 05:08:08.946316 22947 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:08.946353 22947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:08.946439 22947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:08.947381 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data/instance:
uuid: "e91668561b1a4d4bbc92c448c147667f"
format_stamp: "Formatted at 2026-09-24 05:08:08 on dist-test-slave-m01t"
server_key: "81bbc9f6dcb0590810518581b876b6bb"
server_key_iv: "05652ec369c63adf54c487af5adb117f"
server_key_version: "encryptionkey@0"
I20260924 05:08:08.947669 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal/instance:
uuid: "e91668561b1a4d4bbc92c448c147667f"
format_stamp: "Formatted at 2026-09-24 05:08:08 on dist-test-slave-m01t"
server_key: "81bbc9f6dcb0590810518581b876b6bb"
server_key_iv: "05652ec369c63adf54c487af5adb117f"
server_key_version: "encryptionkey@0"
I20260924 05:08:08.948753 22947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:08:08.949347 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:08.949498 22947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:08.949574 22947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal
uuid: "e91668561b1a4d4bbc92c448c147667f"
format_stamp: "Formatted at 2026-09-24 05:08:08 on dist-test-slave-m01t"
server_key: "81bbc9f6dcb0590810518581b876b6bb"
server_key_iv: "05652ec369c63adf54c487af5adb117f"
server_key_version: "encryptionkey@0"
I20260924 05:08:08.949627 22947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:08.955475 22947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:08.956934 22947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:08.957026 22947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:08.960430 22947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45747
I20260924 05:08:08.960467 23014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45747 every 8 connection(s)
I20260924 05:08:08.960738 22947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:08.961606 23015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:08.963043 23015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:08:08 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:08.970402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 22947
I20260924 05:08:08.970499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:08.970685 392 external_mini_cluster.cc:1428] Setting key ab91e3dcf69a73223a7bafab925c9c91
I20260924 05:08:08.971216 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:08.963591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45747 (local address 127.0.0.1:51318)
0924 05:08:08.963753 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:08.963785 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:08.964198 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:08.964405 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:08.964407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:08.964629 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:08.964770 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:08.964774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:08.965670 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:08.965671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:08.966350 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:08.966351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:08.966780 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:08.966781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:08.966837 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:08.967229 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:08.967236 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:08.968475 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:08.969975 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:08.969979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:08.969981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:08.970143 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:08.970340 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:08.970341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:08.970341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:08.970403 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:08.970792 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:08.970806 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:08.970862 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 05:08:08.970929 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:08:08.971047 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260924 05:08:08.971226 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:08.963629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.0.98.62:45747)
0924 05:08:08.963816 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:08.963819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:08.963840 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:08:08.964306 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:08.964308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:08.964323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:08.964392 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:08.964832 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:08.965604 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:08.966408 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:08.966697 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:08.966769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:08.968541 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:08.968547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:08.968547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:08.968553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:08.969912 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:08.970195 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:08.970197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:08.970197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:08.970284 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:08.970452 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:08.970453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:08.970453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:08.970722 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:08.970798 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:08.970970 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 05:08:08.971116 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":167,"thread_start_us":63,"threads_started":1}
I20260924 05:08:08.971871 23015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:08.972190 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:08.972086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38830)
0924 05:08:08.972133 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:08.972172 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 05:08:08.972338 22964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
I20260924 05:08:08.972713 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:08.972640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38842)
0924 05:08:08.972676 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:08.972699 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:08:08.972802 23015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46119: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:08:08.990150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46119
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45747,127.0.98.61:46119,127.0.98.60:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:08:09.022022 23015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } attempt: 1
I20260924 05:08:09.022482 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.022352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38858)
0924 05:08:09.022409 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.022442 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 05:08:09.022855 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.022769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38874)
0924 05:08:09.022813 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.022837 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 05:08:09.022936 23015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46119: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
W20260924 05:08:09.067704 23022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:08:09.067902 23022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:09.067919 23022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:09.067938 23022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:09.069049 23022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:09.069087 23022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:09.069099 23022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:09.069123 23022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:09.069147 23022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:09.070614 23022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45747,127.0.98.61:46119,127.0.98.60:35797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:09.070832 23022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:09.071023 23022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:09.075479 23030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:09.075438 23027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:09.075448 23028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:09.075488 23022 server_base.cc:1034] running on GCE node
I20260924 05:08:09.075804 23022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:09.076079 23022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:09.077219 23022 hybrid_clock.cc:648] HybridClock initialized: now 1790226489077197 us; error 38 us; skew 500 ppm
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:09.079586 23022 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:08:09.079996 23022 webserver.cc:460] Webserver started at http://127.0.98.61:32987/ using document root <none> and password file <none>
I20260924 05:08:09.080180 23022 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:09.080226 23022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:09.080344 23022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:09.081341 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data/instance:
uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6"
format_stamp: "Formatted at 2026-09-24 05:08:09 on dist-test-slave-m01t"
server_key: "cf7e1ee53d3416488faf2863f6e84f22"
server_key_iv: "ccec3742149e07277190881707ecc719"
server_key_version: "encryptionkey@0"
I20260924 05:08:09.081650 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal/instance:
uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6"
format_stamp: "Formatted at 2026-09-24 05:08:09 on dist-test-slave-m01t"
server_key: "cf7e1ee53d3416488faf2863f6e84f22"
server_key_iv: "ccec3742149e07277190881707ecc719"
server_key_version: "encryptionkey@0"
I20260924 05:08:09.082836 23022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:08:09.083433 23037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:09.083573 23022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:09.083650 23022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal
uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6"
format_stamp: "Formatted at 2026-09-24 05:08:09 on dist-test-slave-m01t"
server_key: "cf7e1ee53d3416488faf2863f6e84f22"
server_key_iv: "ccec3742149e07277190881707ecc719"
server_key_version: "encryptionkey@0"
I20260924 05:08:09.083711 23022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:09.091056 23015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } attempt: 2
I20260924 05:08:09.091389 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.091288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38890)
0924 05:08:09.091336 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.091363 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:08:09.091666 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.091593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38896)
0924 05:08:09.091626 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.091648 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 05:08:09.091737 23015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46119: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46119: connect: Connection refused (error 111)
I20260924 05:08:09.097209 23022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:09.098826 23022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:09.098944 23022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:09.102578 23022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46119
I20260924 05:08:09.102617 23089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46119 every 8 connection(s)
I20260924 05:08:09.102916 23022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:08:09.103787 23090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:09.103861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 23022
I20260924 05:08:09.103955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-1/wal/instance
I20260924 05:08:09.104149 392 external_mini_cluster.cc:1428] Setting key e55434cf171e3c62a5850249dcc26508
I20260924 05:08:09.105749 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:09.116490 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.106407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.0.98.62:45747)
0924 05:08:09.106457 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.106459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.106624 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:08:09.107060 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.107061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.107075 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.107144 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.107757 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.108520 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.109481 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.109825 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.109899 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.113935 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.113936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.113936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.113944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.115248 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.115595 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.115596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.115596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.115656 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.115861 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.115862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.115862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.115980 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.116033 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.116313 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.116396 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:08:09.116667 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.106352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45747 (local address 127.0.0.1:51320)
0924 05:08:09.106501 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.106542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.106972 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.107209 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.107212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.107467 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.107666 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.107671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.108599 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.108600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.109401 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.109403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.109907 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.109908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.110009 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.110500 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.110508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.113845 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.115304 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.115311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.115312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.115521 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.115710 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.115711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.115711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.115806 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.116032 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.116038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.116187 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.116284 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.116438 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260924 05:08:09.117202 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:09.125202 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.117492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38906)
0924 05:08:09.117532 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.117542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.117579 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.117832 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.117833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.117977 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.118061 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.118063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.119315 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.119315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.119806 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.119807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.120445 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.120446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.120497 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.120839 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.120842 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.122594 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.124228 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.124229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.124229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.124371 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.124538 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.124538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.124539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.124591 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.124894 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.124894 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.124932 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.124991 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.125091 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 05:08:09.125262 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.117553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.0.98.61:46119)
0924 05:08:09.117686 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.117690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.117707 (+ 17us) server_negotiation.cc:407] Connection header received
0924 05:08:09.117748 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.117750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.117762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.117818 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.118158 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.119241 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.119884 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.120376 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.120453 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.122676 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.122683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.122684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.122690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.124153 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.124433 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.124436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.124436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.124486 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.124638 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.124639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.124639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.124834 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.124908 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.125020 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.125140 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:08:09.126032 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:09.126379 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.126247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47194)
0924 05:08:09.126300 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.126351 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:08:09.126513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyORENn/build/release/bin/kudu
/tmp/dist-test-taskyORENn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35797
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45747,127.0.98.61:46119,127.0.98.60:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:09.126520 23041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
I20260924 05:08:09.126940 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.126845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47204)
0924 05:08:09.126888 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.126917 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 05:08:09.127076 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35797: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
I20260924 05:08:09.176316 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } attempt: 1
I20260924 05:08:09.176815 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.176681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47208)
0924 05:08:09.176739 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.176779 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 05:08:09.177110 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.177043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47224)
0924 05:08:09.177074 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.177095 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 05:08:09.177198 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35797: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
I20260924 05:08:09.182875 23015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } attempt: 3
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:09.189476 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.183132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38918)
0924 05:08:09.183192 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.183202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.183250 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.183382 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.183383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.183530 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.183592 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.183593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.184231 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.184232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.184745 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.184747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.185095 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.185096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.185165 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.185551 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.185556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.187000 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.188552 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.188553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.188554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.188720 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.188891 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.188892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.188892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.188949 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.189165 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.189166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.189214 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.189273 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.189378 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 05:08:09.189473 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.183192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.0.98.61:46119)
0924 05:08:09.183244 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.183245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.183255 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:09.183311 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.183312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.183324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.183371 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.183663 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.184147 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.184824 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.185034 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.185090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.187102 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.187104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.187104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.187110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.188474 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.188788 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.188789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.188789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.188840 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.188998 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.188999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.189000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.189107 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.189161 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.189301 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.189378 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:08:09.189860 23015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:09.190217 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.190117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47234)
0924 05:08:09.190161 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.190192 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:08:09.190457 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.190401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47236)
0924 05:08:09.190425 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.190445 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 05:08:09.190555 23015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35797: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
W20260924 05:08:09.206300 23097 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:08:09.206496 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:09.206521 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:09.206553 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:09.207657 23097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:09.207691 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:09.207710 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:09.207726 23097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:09.207749 23097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:09.209163 23097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45747,127.0.98.61:46119,127.0.98.60:35797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:09.209385 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:09.209581 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:09.214001 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:09.214018 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:09.214164 23097 server_base.cc:1034] running on GCE node
W20260924 05:08:09.214030 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:09.214432 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:09.214674 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:09.215802 23097 hybrid_clock.cc:648] HybridClock initialized: now 1790226489215789 us; error 27 us; skew 500 ppm
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:09.218051 23097 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:08:09.218479 23097 webserver.cc:460] Webserver started at http://127.0.98.60:35959/ using document root <none> and password file <none>
I20260924 05:08:09.218662 23097 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:09.218699 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:09.218786 23097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:09.219731 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data/instance:
uuid: "d9c021e994574dacb320636fc0f3002b"
format_stamp: "Formatted at 2026-09-24 05:08:09 on dist-test-slave-m01t"
server_key: "793e7632baf2227356f022409bdf49f8"
server_key_iv: "8c7d86ffdb68dde533dd344eb990906d"
server_key_version: "encryptionkey@0"
I20260924 05:08:09.220003 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal/instance:
uuid: "d9c021e994574dacb320636fc0f3002b"
format_stamp: "Formatted at 2026-09-24 05:08:09 on dist-test-slave-m01t"
server_key: "793e7632baf2227356f022409bdf49f8"
server_key_iv: "8c7d86ffdb68dde533dd344eb990906d"
server_key_version: "encryptionkey@0"
I20260924 05:08:09.221114 23097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:08:09.221753 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:09.221683 23015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } attempt: 1
I20260924 05:08:09.221923 23097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:09.222007 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal
uuid: "d9c021e994574dacb320636fc0f3002b"
format_stamp: "Formatted at 2026-09-24 05:08:09 on dist-test-slave-m01t"
server_key: "793e7632baf2227356f022409bdf49f8"
server_key_iv: "8c7d86ffdb68dde533dd344eb990906d"
server_key_version: "encryptionkey@0"
I20260924 05:08:09.222057 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.221945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47250)
0924 05:08:09.221989 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.222026 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:08:09.222069 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:09.222323 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.222233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47258)
0924 05:08:09.222270 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.222296 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:08:09.222430 23015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35797: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35797: connect: Connection refused (error 111)
I20260924 05:08:09.235391 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:09.237594 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:09.237743 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:09.241344 23097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35797
I20260924 05:08:09.241379 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35797 every 8 connection(s)
I20260924 05:08:09.241704 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:08:09.242694 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:09.244192 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:09.245328 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } attempt: 2
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:09.250993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyORENn/build/release/bin/kudu as pid 23097
I20260924 05:08:09.251103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyORENn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226228698807-392-0/minicluster-data/master-2/wal/instance
I20260924 05:08:09.251369 392 external_mini_cluster.cc:1428] Setting key 53145c1890d808597cda086ab1f563d2
I20260924 05:08:09.252702 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.244880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.0.98.62:45747)
0924 05:08:09.244959 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.244961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.245081 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:08:09.245571 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.245572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.245583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.245626 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.246311 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.246771 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.247596 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.247775 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.247829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.249807 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.249808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.249808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.249815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.251196 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.251739 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.251740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.251741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.251800 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.252072 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.252073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.252074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.252208 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.252266 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.252578 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.252641 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 05:08:09.252928 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.244791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45747 (local address 127.0.0.1:51334)
0924 05:08:09.244961 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.245005 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.245468 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.245851 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.245854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.246078 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.246221 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.246225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.246838 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.246839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.247519 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.247521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.247858 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.247859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.247966 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.248357 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.248365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.249685 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.251270 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.251295 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.251297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.251655 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.251878 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.251879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.251879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.252017 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.252284 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.252289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.252451 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.252564 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.252700 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260924 05:08:09.253410 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:09.253656 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.245561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47260)
0924 05:08:09.245610 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.245619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.245656 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.245963 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.245964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.246111 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.246152 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.246155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.247096 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.247097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.247588 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.247590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.248032 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.248033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.248090 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.248457 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.248462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.250056 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.251890 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.251891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.251892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.252066 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.253033 (+ 967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.253034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.253035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.253102 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.253360 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.253361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.253405 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.253467 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.253562 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:08:09.253664 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.245624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.98.60:35797)
0924 05:08:09.245806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.245810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.245825 (+ 15us) server_negotiation.cc:407] Connection header received
0924 05:08:09.245850 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.245851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.245878 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.245951 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.246232 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.247016 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.247661 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.247968 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.248024 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.250127 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.250138 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.250138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.250145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.251815 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.252133 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.252136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.252136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.252212 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.253161 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.253162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.253163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.253309 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.253379 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.253513 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.253573 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260924 05:08:09.255330 23090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: Bootstrap starting.
I20260924 05:08:09.256181 23090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:09.256456 23090 log.cc:824] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: Log is configured to *not* fsync() on all Append() calls
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:09.257226 23090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: No bootstrap required, opened a new log
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:08:09.259250 23090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.259394 23090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa4e82692ea4beab7e80e7b3a5f1bc6, State: Initialized, Role: FOLLOWER
I20260924 05:08:09.259476 23090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.259735 23172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.259790 23090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:09.260268 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.253605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46119 (local address 127.0.0.1:38928)
0924 05:08:09.254108 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.254117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.254153 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.254277 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.254277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.254399 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.254440 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.254441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.255128 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.255130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.255849 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.255853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.256331 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.256332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.256402 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.256726 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.256730 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.257923 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.259255 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.259255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.259256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.259466 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.259660 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.259660 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.259661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.259717 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.259999 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.260000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.260041 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.260112 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.260182 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490}
I20260924 05:08:09.260289 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.253683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.0.98.61:46119)
0924 05:08:09.253789 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.253790 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.254196 (+ 406us) server_negotiation.cc:407] Connection header received
0924 05:08:09.254214 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.254215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.254222 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.254262 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.254502 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.255047 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.255942 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.256244 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.256312 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.257994 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.257996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.257996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.258003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.259183 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.259541 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.259542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.259542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.259597 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.259780 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.259781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.259781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.259932 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.259997 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.260143 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.260209 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 05:08:09.260675 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:08:09.260928 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.251697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.0.98.60:35797)
0924 05:08:09.251824 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.251826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.252240 (+ 414us) server_negotiation.cc:407] Connection header received
0924 05:08:09.252258 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.252259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.252268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.252325 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.253243 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.253890 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.254794 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.255068 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.255118 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.257694 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.257695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.257695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.257701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.259804 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.260196 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.260197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.260197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.260248 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.260435 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.260436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.260437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.260548 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.260662 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.260736 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.260810 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260924 05:08:09.262638 23183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:08:09.262688 23183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:08:09.261540 23172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:09.262835 23022 master_runner.cc:186] Error getting master registration for 127.0.98.62:45747: OK, instance_id { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" instance_seqno: 1790226488959505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:08:09.265419 23022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:09.268273 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.260896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47264)
0924 05:08:09.260987 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.260994 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.261032 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.261257 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.261258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.261395 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.261430 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.261432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.262146 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.262147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.262876 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.262879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.263316 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.263317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.263360 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.263714 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.263718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.265250 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.266815 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.266816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.266816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.266991 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.267160 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.267161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.267161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.267204 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.267968 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.267968 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.268010 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.268077 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.268150 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 05:08:09.268342 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.260956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.0.98.60:35797)
0924 05:08:09.260992 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.260994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.261179 (+ 185us) server_negotiation.cc:407] Connection header received
0924 05:08:09.261195 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.261196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.261205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.261248 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.261494 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.262081 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.262951 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.263227 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.263278 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.265328 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.265329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.265329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.265334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.266761 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.267060 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.267061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.267061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.267105 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.267258 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.267258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.267258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.267393 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.267451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.268128 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.268214 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:08:09.269223 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.261627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.98.62:45747)
0924 05:08:09.261666 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.261668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.261729 (+ 61us) server_negotiation.cc:407] Connection header received
0924 05:08:09.261798 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.261799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.261808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.261857 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.262304 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.262987 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.263875 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.264147 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.264201 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.266575 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.266576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.266576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.266583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.267888 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.268229 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.268230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.268231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.268284 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.268482 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.268483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.268483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.268597 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.268737 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.269068 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.269136 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:09.269590 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: Bootstrap starting.
I20260924 05:08:09.270200 23165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:09.270445 23165 log.cc:824] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:09.271090 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: No bootstrap required, opened a new log
I20260924 05:08:09.272712 23165 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.272827 23165 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c021e994574dacb320636fc0f3002b, State: Initialized, Role: FOLLOWER
I20260924 05:08:09.272908 23165 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.273140 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.273201 23165 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:09.273257 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260924 05:08:09.276139 23197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:08:09.276186 23197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:08:09.276425 23097 master_runner.cc:186] Error getting master registration for 127.0.98.62:45747: OK, instance_id { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" instance_seqno: 1790226488959505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:08:09.278729 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.269830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.0.98.61:46119)
0924 05:08:09.270658 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.270659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.270669 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:09.270688 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.270688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.270697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.270755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.271320 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.271973 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.273043 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.273331 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.273386 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.275912 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.275913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.275913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.275920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.277381 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.277745 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.277746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.277747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.277807 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.277984 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.277985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.277985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.278097 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.278209 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.278575 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.278640 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806}
I20260924 05:08:09.278775 23097 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:08:09.280015 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:08:09.280776 22989 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e91668561b1a4d4bbc92c448c147667f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51336
I20260924 05:08:09.297577 23015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } attempt: 2
Sep 24 05:08:09 dist-test-slave-m01t krb5kdc[22934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:08:09.305305 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.297908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35797 (local address 127.0.0.1:47268)
0924 05:08:09.297958 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:09.297968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:09.298006 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:09.298199 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:09.298200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:09.298375 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:09.298421 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.298423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.299339 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.299340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.300158 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:09.300160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.300612 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.300613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:09.300673 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.301125 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:09.301130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:09.302577 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:09.304273 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.304274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.304274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.304478 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.304672 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:09.304672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:09.304673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:09.304731 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:09.305009 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:09.305010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:09.305053 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 05:08:09.305117 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:08:09.305205 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:08:09.305311 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.297974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.0.98.60:35797)
0924 05:08:09.298075 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.298077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.298083 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:08:09.298106 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.298106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.298115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.298158 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.298499 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.299255 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.300239 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.300541 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.300606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.302651 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.302653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.302653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.302660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.304195 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.304551 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.304552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.304552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.304614 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.304807 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.304808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.304808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.304943 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.305016 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.305144 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.305218 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 05:08:09.307152 23015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f: Bootstrap starting.
I20260924 05:08:09.307879 23015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:09.308169 23015 log.cc:824] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:09.309023 23015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f: No bootstrap required, opened a new log
I20260924 05:08:09.310423 23015 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.310570 23015 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91668561b1a4d4bbc92c448c147667f, State: Initialized, Role: FOLLOWER
I20260924 05:08:09.310649 23015 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.310953 23015 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:09.310918 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.311087 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:09.312255 22989 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 05:08:09.312602 22989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 0 FOLLOWER]: Advancing to term 1
W20260924 05:08:09.312798 23211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:08:09.312896 23211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:08:09.313695 22989 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.313943 22989 leader_election.cc:288] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [CANDIDATE]: Term 1 election: Requested vote from peers 9aa4e82692ea4beab7e80e7b3a5f1bc6 (127.0.98.61:46119), d9c021e994574dacb320636fc0f3002b (127.0.98.60:35797)
I20260924 05:08:09.314321 23064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e91668561b1a4d4bbc92c448c147667f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6"
I20260924 05:08:09.314472 23064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:09.315028 23139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e91668561b1a4d4bbc92c448c147667f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d9c021e994574dacb320636fc0f3002b"
I20260924 05:08:09.315160 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:09.315678 23064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91668561b1a4d4bbc92c448c147667f in term 1.
I20260924 05:08:09.316011 23139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91668561b1a4d4bbc92c448c147667f in term 1.
I20260924 05:08:09.316250 22966 leader_election.cc:302] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c021e994574dacb320636fc0f3002b, e91668561b1a4d4bbc92c448c147667f; no voters:
I20260924 05:08:09.316385 23200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:09.316533 23200 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [term 1 LEADER]: Becoming Leader. State: Replica: e91668561b1a4d4bbc92c448c147667f, State: Running, Role: LEADER
I20260924 05:08:09.316613 23200 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } }
I20260924 05:08:09.317026 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91668561b1a4d4bbc92c448c147667f. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.317091 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:09.317828 22947 master_runner.cc:263] No leader master found from master e91668561b1a4d4bbc92c448c147667f
I20260924 05:08:09.317878 22947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:08:09.327939 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.316597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.98.62:45747)
0924 05:08:09.316633 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.316635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.317190 (+ 555us) server_negotiation.cc:407] Connection header received
0924 05:08:09.317307 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.317308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.317319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.317349 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.318589 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.319326 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.320363 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.320663 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.320703 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.323142 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.323143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.323144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.323152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.324770 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.325238 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.325239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.325240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.325296 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.327319 (+ 2023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.327320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.327320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.327479 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.327555 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.327761 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.327839 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:08:09.328729 23221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:09.328962 23221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:09.334530 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.316934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.0.98.60:35797)
0924 05:08:09.316967 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.316968 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.322428 (+ 5460us) server_negotiation.cc:407] Connection header received
0924 05:08:09.322522 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.322524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.322535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.322563 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.325523 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.326308 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.329802 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.330119 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.330160 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.331680 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.331681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.331682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.331690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.333242 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.333638 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.333639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.333640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.333694 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.333932 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.333933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.333933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.334070 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.334140 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.334339 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.334430 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:08:09.339608 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:09.328438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.61:46119)
0924 05:08:09.328480 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.328482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.328711 (+ 229us) server_negotiation.cc:407] Connection header received
0924 05:08:09.328871 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.328872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.328883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.328911 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.329639 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.330318 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.334892 (+ 4574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:09.335172 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:09.335219 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:09.336649 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:09.336650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:09.336651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:09.336659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:09.338257 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.338775 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.338776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.338776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.338833 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:09.339023 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:09.339024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:09.339024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:09.339153 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:09.339221 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:09.339431 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 05:08:09.339507 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:08:09.340752 23139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [term 1 FOLLOWER]: Refusing update from remote peer e91668561b1a4d4bbc92c448c147667f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:09.341085 23217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:09.342442 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91668561b1a4d4bbc92c448c147667f. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.342525 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:09.342861 23064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [term 1 FOLLOWER]: Refusing update from remote peer e91668561b1a4d4bbc92c448c147667f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:09.343115 23217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:09.344103 23172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91668561b1a4d4bbc92c448c147667f. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.344188 23172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:08:09.349444 23043 tablet.cc:2369] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:08:09.350517 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.350593 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:09.350627 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.350661 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:09.350715 23172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.350800 23172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:09.351120 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e91668561b1a4d4bbc92c448c147667f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91668561b1a4d4bbc92c448c147667f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45747 } } peers { permanent_uuid: "9aa4e82692ea4beab7e80e7b3a5f1bc6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46119 } } peers { permanent_uuid: "d9c021e994574dacb320636fc0f3002b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35797 } } }
I20260924 05:08:09.351192 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:08:09.351425 23224 mvcc.cc:205] Tried to move back new op lower bound from 7332767700336848896 to 7332767700241600512. Current Snapshot: MvccSnapshot[applied={T|T < 7332767700336848896}]
I20260924 05:08:09.351435 23223 mvcc.cc:205] Tried to move back new op lower bound from 7332767700336848896 to 7332767700241600512. Current Snapshot: MvccSnapshot[applied={T|T < 7332767700336848896}]
I20260924 05:08:09.351533 23226 mvcc.cc:205] Tried to move back new op lower bound from 7332767700336848896 to 7332767700241600512. Current Snapshot: MvccSnapshot[applied={T|T < 7332767700336848896}]
I20260924 05:08:09.354657 23221 catalog_manager.cc:1347] Generated new cluster ID: 80642d8f39a2431fa4c1772390c4e012
I20260924 05:08:09.354698 23221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:09.363608 23221 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:09.364293 23221 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:09.372678 23221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e91668561b1a4d4bbc92c448c147667f: Generated new TSK 0
I20260924 05:08:09.372884 23221 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:08:09.397846 23018 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.396179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.98.62:45747)
0924 05:08:09.396221 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.396223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.397154 (+ 931us) server_negotiation.cc:407] Connection header received
0924 05:08:09.397284 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.397285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.397296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.397327 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.397778 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51360: BlockingRecv error: recv got EOF from 127.0.0.1:51360 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:08:09.398558 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.396521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.0.98.60:35797)
0924 05:08:09.396558 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.396560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.398074 (+ 1514us) server_negotiation.cc:407] Connection header received
0924 05:08:09.398166 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.398167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.398176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.398203 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.398501 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47296: BlockingRecv error: recv got EOF from 127.0.0.1:47296 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 05:08:09.399262 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.396879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.98.61:46119)
0924 05:08:09.396917 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.396919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.398783 (+ 1864us) server_negotiation.cc:407] Connection header received
0924 05:08:09.398876 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.398877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.398887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.398914 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.399204 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38958: BlockingRecv error: recv got EOF from 127.0.0.1:38958 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:08:09.401381 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.400699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.0.98.61:46119)
0924 05:08:09.400731 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.400733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.400922 (+ 189us) server_negotiation.cc:407] Connection header received
0924 05:08:09.401016 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.401017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.401032 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.401057 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.401342 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38974: BlockingRecv error: recv got EOF from 127.0.0.1:38974 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 05:08:09.404179 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.403484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.0.98.61:46119)
0924 05:08:09.403513 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.403515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.403723 (+ 208us) server_negotiation.cc:407] Connection header received
0924 05:08:09.403824 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.403824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.403834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.403859 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.404138 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38982: BlockingRecv error: recv got EOF from 127.0.0.1:38982 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:08:09.407866 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.407171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.0.98.61:46119)
0924 05:08:09.407202 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.407203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.407413 (+ 210us) server_negotiation.cc:407] Connection header received
0924 05:08:09.407498 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.407499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.407507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.407533 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.407826 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38992: BlockingRecv error: recv got EOF from 127.0.0.1:38992 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 05:08:09.413568 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.412911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.0.98.61:46119)
0924 05:08:09.412940 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.412942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.413140 (+ 198us) server_negotiation.cc:407] Connection header received
0924 05:08:09.413220 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.413221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.413229 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.413254 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.413525 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39006: BlockingRecv error: recv got EOF from 127.0.0.1:39006 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:08:09.423647 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.422834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.0.98.61:46119)
0924 05:08:09.422877 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.422879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.423124 (+ 245us) server_negotiation.cc:407] Connection header received
0924 05:08:09.423210 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.423211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.423222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.423247 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.423589 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39010: BlockingRecv error: recv got EOF from 127.0.0.1:39010 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:08:09.441497 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.440858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.0.98.61:46119)
0924 05:08:09.440901 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.440904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.441000 (+ 96us) server_negotiation.cc:407] Connection header received
0924 05:08:09.441064 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.441065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.441076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.441117 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.441438 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39018: BlockingRecv error: recv got EOF from 127.0.0.1:39018 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:08:09.475248 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.474539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.0.98.61:46119)
0924 05:08:09.474593 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.474596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.474730 (+ 134us) server_negotiation.cc:407] Connection header received
0924 05:08:09.474804 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.474805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.474818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.474864 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.475187 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39034: BlockingRecv error: recv got EOF from 127.0.0.1:39034 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 05:08:09.540877 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.540336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.0.98.61:46119)
0924 05:08:09.540391 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.540393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.540451 (+ 58us) server_negotiation.cc:407] Connection header received
0924 05:08:09.540512 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.540512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.540523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.540571 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.540831 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39040: BlockingRecv error: recv got EOF from 127.0.0.1:39040 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 05:08:09.670648 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.670071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.0.98.61:46119)
0924 05:08:09.670129 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.670131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.670209 (+ 78us) server_negotiation.cc:407] Connection header received
0924 05:08:09.670263 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.670264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.670274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.670330 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.670599 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39044: BlockingRecv error: recv got EOF from 127.0.0.1:39044 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 05:08:09.928355 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:08:09.927801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.98.61:46119)
0924 05:08:09.927857 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:09.927858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:09.927872 (+ 14us) server_negotiation.cc:407] Connection header received
0924 05:08:09.927936 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:09.927936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:09.927947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:09.928006 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:09.928305 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39060: BlockingRecv error: recv got EOF from 127.0.0.1:39060 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:08:10.263087 23183 catalog_manager.cc:1259] Loaded cluster ID: 80642d8f39a2431fa4c1772390c4e012
I20260924 05:08:10.263160 23183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: loading cluster ID for follower catalog manager: success
I20260924 05:08:10.263936 23183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: acquiring CA information for follower catalog manager: success
I20260924 05:08:10.264221 23183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9aa4e82692ea4beab7e80e7b3a5f1bc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:08:10.276535 23197 catalog_manager.cc:1259] Loaded cluster ID: 80642d8f39a2431fa4c1772390c4e012
I20260924 05:08:10.276590 23197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: loading cluster ID for follower catalog manager: success
I20260924 05:08:10.277359 23197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: acquiring CA information for follower catalog manager: success
I20260924 05:08:10.277612 23197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9c021e994574dacb320636fc0f3002b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:08:10.443722 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:10.441527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.0.98.61:46119)
0924 05:08:10.441632 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:10.441634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:10.441645 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:08:10.441696 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:10.441697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:10.441765 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:10.441846 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:10.441963 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.442452 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.443132 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.443313 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.443353 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:10.443467 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:10.443569 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:10.443621 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:10.443639 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:08:10.443655 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260924 05:08:10.447326 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:10.445312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.0.98.60:35797)
0924 05:08:10.445398 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:10.445400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:10.445406 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:08:10.445440 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:10.445441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:10.445472 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:10.445534 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:10.445606 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.446105 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.446744 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.446926 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.446968 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:10.447059 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:10.447146 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:10.447194 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:10.447209 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 05:08:10.447223 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":46,"threads_started":1}
I20260924 05:08:10.451440 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:10.448713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.61:46119)
0924 05:08:10.448807 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:10.448809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:10.448906 (+ 97us) server_negotiation.cc:407] Connection header received
0924 05:08:10.448984 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:10.448985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:10.448996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:10.449048 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:10.449285 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.449757 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.450715 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.450938 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.450998 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:10.451116 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:10.451142 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:10.451188 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:10.451350 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 05:08:10.451370 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 05:08:10.451551 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:10.448892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.0.98.60:35797)
0924 05:08:10.448927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:10.448929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:10.449046 (+ 117us) server_negotiation.cc:407] Connection header received
0924 05:08:10.449075 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:10.449076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:10.449085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:10.449124 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:10.449243 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.449966 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.450884 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.451137 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.451191 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:10.451311 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:10.451334 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:10.451375 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:10.451469 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 05:08:10.451488 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:08:10.451953 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:10.448714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.98.62:45747)
0924 05:08:10.448819 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:10.448821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:10.448829 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:08:10.448953 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:10.448954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:10.449000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:10.449067 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:10.449206 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.449988 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.451032 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:10.451327 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:10.451382 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:10.451594 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:10.451725 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:10.451786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:10.451860 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:08:10.451882 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260924 05:08:10.453364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 22947
I20260924 05:08:10.457916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 23022
I20260924 05:08:10.461875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyORENn/build/release/bin/kudu with pid 23097
2026-09-24T05:08:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2669 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5300 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (261767 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST